Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BM25 › BM25 Variants

Michael BrenndoerferUpdated March 21, 20266 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 14BM25 › BM25 Variants

Reading section 7 of 14 · BM25 Variants · 6 minSection 7 of 14 · 6 min

Section 7 of 14 · 6 min read

BM25 › BM25 VariantsOne section · Sections in this chapter7 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The original BM25 formula has some known issues that researchers have addressed with variants. Understanding these variants helps you make better choices when the default BM25 underperforms in specific situations.

The Problem with Short DocumentsLink Copied

One issue with BM25 is that very short documents can be over-penalized. If a document is much shorter than average, the length normalization term $1 - b + b \cdot \frac{|D|}{\text{avgdl}}$ becomes less than 1, which boosts the term frequency component. But this boost might not be enough to compensate for the document simply having fewer terms.

Consider a collection of academic abstracts with an average length of 200 words, but you've added some very short 20-word author bio entries. Those bio entries might contain a query term once, but because they're so short relative to the average, the length normalization boost might make them score oddly high. More seriously, the normalization approaches break down mathematically when documents are extremely short.

More fundamentally, there's a theoretical concern: Lv and Zhai (2011) showed that the original BM25 violates a property called the "term frequency normalization lower bound." This property says that a document containing a query term should always score higher than the same document without the query term. At extreme length differences, standard BM25 can violate this. The variants below fix this.

BM25L: Lower Bound on TFLink Copied

BM25L (BM25 with Lower bound) addresses this by modifying how term frequency is normalized. First, it computes a length-normalized term frequency:

c(t, D) = \frac{f(t, D)}{1 - b + b \cdot \frac{|D|}{\text{avgdl}}}

where:

$c(t, D)$ : the length-normalized term frequency
$f(t, D)$ : raw frequency of term $t$ in document $D$
$b$ : length normalization parameter (typically 0.75)
$|D|$ : length of document $D$ (number of terms)
$\text{avgdl}$ : average document length across the collection

This formula essentially "inflates" the term frequency for short documents and "deflates" it for long documents. A document half the average length would have its term frequency doubled; a document twice the average length would have it halved.

Then BM25L adds a lower bound constant $\delta$ to this normalized value:

\text{BM25L}(D, Q) = \sum_{t \in Q} \text{IDF}(t) \cdot \frac{(k_1 + 1) \cdot (c(t, D) + \delta)}{k_1 + c(t, D) + \delta}

where:

$D$ : the document being scored
$Q$ : the query (set of query terms)
$t$ : a term in the query
$\text{IDF}(t)$ : inverse document frequency of term $t$
$c(t, D)$ : length-normalized term frequency (defined above)
$\delta$ : lower bound constant (typically 0.5)
$k_1$ : saturation parameter (same as in standard BM25)

The addition of $\delta$ to the normalized term frequency ensures that even a single occurrence of a term makes a nonzero contribution to the score, regardless of document length. This $\delta$ acts as a guarantee: the term frequency component can never fall below $\delta/(k_1 + \delta)$ , which is strictly positive. This restores the lower bound property that standard BM25 can violate.

BM25L

A variant of BM25 that adds a small constant $\delta$ to the length-normalized term frequency, preventing the term frequency normalization lower bound from being violated and ensuring consistent behavior across documents of very different lengths.

BM25+: Additive IDFLink Copied

BM25+ takes a different approach by adding a constant to the entire term weight:

\text{BM25+}(D, Q) = \sum_{t \in Q} \text{IDF}(t) \cdot \left(\frac{f(t, D) \cdot (k_1 + 1)}{f(t, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)} + \delta\right)

where:

$D$ : the document being scored
$Q$ : the query (set of query terms)
$t$ : a term in the query
$\text{IDF}(t)$ : inverse document frequency of term $t$
$f(t, D)$ : frequency of term $t$ in document $D$
$k_1$ : saturation parameter (typically 1.2 to 2.0)
$b$ : length normalization parameter (typically 0.75)
$|D|$ : length of document $D$ (number of terms)
$\text{avgdl}$ : average document length in the collection
$\delta$ : additive constant (typically 1.0)

The key insight is that $\delta$ is added inside the IDF multiplication. This guarantees that a document containing a query term always scores higher than one that doesn't, regardless of length normalization effects. Even if the term frequency component becomes very small due to aggressive length normalization, the $\delta$ ensures a minimum positive contribution. The IDF scaling means that this floor is higher for rare terms (where it matters more) than for common terms.

Let's compare these variants:

In[23]:

Code

def bm25_standard(tf, idf, doc_len, avgdl, k1=1.2, b=0.75):
    """Standard BM25 scoring."""
    numerator = tf * (k1 + 1)
    denominator = tf + k1 * (1 - b + b * (doc_len / avgdl))
    return idf * (numerator / denominator)


def bm25l(tf, idf, doc_len, avgdl, k1=1.2, b=0.75, delta=0.5):
    """BM25L with lower bound on normalized TF."""
    # First normalize TF by document length
    ctd = tf / (1 - b + b * (doc_len / avgdl))
    # Then add delta to the normalized value
    ctd_adjusted = ctd + delta
    numerator = (k1 + 1) * ctd_adjusted
    denominator = k1 + ctd_adjusted
    return idf * (numerator / denominator)


def bm25_plus(tf, idf, doc_len, avgdl, k1=1.2, b=0.75, delta=1.0):
    """BM25+ with additive component."""
    numerator = tf * (k1 + 1)
    denominator = tf + k1 * (1 - b + b * (doc_len / avgdl))
    return idf * ((numerator / denominator) + delta)


# Compare for different document lengths
avgdl = 100
tf = 2
idf = 2.0  # Moderately rare term
doc_lengths = [20, 50, 100, 200, 500]

comparison = {"BM25": [], "BM25L": [], "BM25+": []}

for doc_len in doc_lengths:
    comparison["BM25"].append(bm25_standard(tf, idf, doc_len, avgdl))
    comparison["BM25L"].append(bm25l(tf, idf, doc_len, avgdl))
    comparison["BM25+"].append(bm25_plus(tf, idf, doc_len, avgdl))

Out[24]:

Console

| Doc Length | BM25 | BM25L | BM25+ |
|---:|---:|---:|---:|
| 20 | 3.55 | 3.61 | 5.55 |
| 50 | 3.20 | 3.32 | 5.20 |
| 100 | 2.75 | 2.97 | 4.75 |
| 200 | 2.15 | 2.54 | 4.15 |
| 500 | 1.29 | 2.00 | 3.29 |

: BM25 variant scores across document lengths (TF=2, avgdl=100, IDF=2.0). BM25+ maintains higher scores due to its additive delta term. {#tbl-bm25-variants}

BM25+ consistently gives higher scores, especially for shorter documents where the additive $\delta$ term has more relative impact. BM25L provides a middle ground with its adjusted term frequency calculation. For most practical applications, the differences are small, and the defaults work well. The variants matter primarily when your collection has a mix of very short and very long documents, such as a mix of social media posts and full articles.