Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

TF-IDF › BM25 as a TF-IDF Extension

Michael BrenndoerferUpdated March 21, 20266 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 13TF-IDF › BM25 as a TF-IDF Extension

Reading section 9 of 13 · BM25 as a TF-IDF Extension · 6 minSection 9 of 13 · 6 min

Section 9 of 13 · 6 min read

TF-IDF › BM25 as a TF-IDF ExtensionOne section · Sections in this chapter9 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

TF-IDF is powerful, but it has a known weakness: term frequency saturation. In the basic formula, a document where "neural" appears 50 times scores 50 times higher on that term than a document where it appears once, even though the information gain from each additional repetition diminishes rapidly. If a document mentions "deep learning" twice in a focused 200-word post, and another document mentions it 40 times in a rambling 5,000-word discussion, TF-IDF would heavily favor the longer document even though the shorter one might be more focused and relevant.

BM25 (Best Match 25), developed in the 1990s by Stephen Robertson and colleagues at City University London, addresses this through a more principled probabilistic framework. It builds on TF-IDF's core intuition but adds two structural improvements: a saturation function that limits the TF contribution, and explicit length normalization that penalizes long documents proportionally to their length.

How BM25 Differs from TF-IDFLink Copied

BM25 replaces the simple $\text{tf} \times \text{idf}$ multiplication with a more complex scoring function:

\text{BM25}(t, d, D) = \text{idf}(t, D) \times \frac{f_{t,d} \cdot (k_1 + 1)}{f_{t,d} + k_1 \cdot \left(1 - b + b \cdot \frac{|d|}{\text{avgdl}}\right)}

where:

$f_{t,d}$ : raw frequency of term $t$ in document $d$
$k_1$ : saturation parameter, typically 1.2 to 2.0, controlling how quickly the TF contribution saturates. Lower values saturate faster.
$b$ : length normalization parameter, typically 0.75, controlling how much document length affects scoring. Setting $b=0$ disables length normalization entirely; $b=1$ applies full normalization proportional to document length.
$|d|$ : number of tokens in document $d$
$\text{avgdl}$ : average document length across the corpus
$\text{idf}(t, D)$ : IDF component, often using the BM25-specific formula $\log\!\left(\frac{N - \text{df}(t) + 0.5}{\text{df}(t) + 0.5} + 1\right)$

The saturation effect works through the denominator. As $f_{t,d}$ grows, the TF contribution approaches an asymptotic ceiling. Specifically, in the limit where $f_{t,d} \to \infty$ and holding the length normalization term constant at some value $L$ :

\frac{f_{t,d} \cdot (k_1 + 1)}{f_{t,d} + k_1 \cdot L} \to \frac{k_1 + 1}{1} = k_1 + 1

This means the TF contribution can never exceed $k_1 + 1$ regardless of how many times the term appears. With $k_1 = 1.5$ , the ceiling is 2.5. A term appearing 3 times and a term appearing 100 times both contribute at most 2.5 to the score (and the difference between them is small). This prevents keyword stuffing from artificially inflating scores.

The length normalization term $1 - b + b \cdot \frac{|d|}{\text{avgdl}}$ is 1 when the document is exactly average length, more than 1 for longer documents, and less than 1 for shorter documents. Adding this to the denominator means longer documents have a larger denominator, which reduces their TF contribution proportionally. A document twice the average length gets its TF contributions halved (approximately), correcting for the inflated raw counts that longer documents naturally accumulate.

TF Saturation VisualizationLink Copied

The saturation effect is the most important conceptual difference between TF-IDF and BM25.

Out[32]:

Visualization

Line plot comparing TF-IDF linear TF contribution versus BM25 saturating TF contribution across term frequencies 0 to 30. — Term frequency contribution under TF-IDF (linear) versus BM25 (saturating) as raw count increases from 0 to 30. TF-IDF grows without bound, meaning that very high term counts can dominate scores disproportionately. BM25 with k1=1.2 approaches a ceiling of 2.2, while k1=2.0 approaches a ceiling of 3.0, which ensures diminishing returns for repetition.

The saturation curves make the key difference viscerally clear. Under TF-IDF, a term appearing 30 times scores 30 times higher than a term appearing once. Under BM25 with $k_1 = 1.2$ , the ratio is only about 1.9:1. This is the saturation property in action: the first occurrence of a term provides the most information, and each subsequent occurrence provides less. BM25's curve shape captures this diminishing-returns intuition directly in the formula.

Implementing BM25Link Copied

In[33]:

Code

class BM25:
    """Minimal BM25 implementation for illustration."""

    def __init__(self, k1=1.5, b=0.75):
        self.k1 = k1
        self.b = b

    def fit(self, tokenized_corpus):
        self.corpus = tokenized_corpus
        self.N = len(tokenized_corpus)
        self.avgdl = np.mean([len(doc) for doc in tokenized_corpus])

        # Document frequency
        self.df = Counter()
        for doc in tokenized_corpus:
            self.df.update(set(doc))

        # BM25 IDF
        self.idf = {}
        for term, df_t in self.df.items():
            self.idf[term] = np.log((self.N - df_t + 0.5) / (df_t + 0.5) + 1)
        return self

    def score(self, query_tokens, doc_tokens):
        """BM25 score for a query against a single document."""
        tf = Counter(doc_tokens)
        dl = len(doc_tokens)
        score = 0.0
        for term in query_tokens:
            if term not in self.idf:
                continue
            f = tf.get(term, 0)
            idf_t = self.idf[term]
            numerator = f * (self.k1 + 1)
            denominator = f + self.k1 * (1 - self.b + self.b * dl / self.avgdl)
            score += idf_t * (numerator / denominator)
        return score

    def rank(self, query_tokens):
        """Rank all documents in the corpus for a query."""
        scores = [self.score(query_tokens, doc) for doc in self.corpus]
        return sorted(enumerate(scores), key=lambda x: x[1], reverse=True)


bm25 = BM25(k1=1.5, b=0.75)
bm25.fit(tokenized_docs)

Out[34]:

Console

Query: 'deep neural networks for image tasks'

BM25 ranking:
  1. Doc 2 (score=5.0219): Deep learning uses neural networks. Neural networks lea...
  2. Doc 4 (score=2.3763): Computer vision analyzes images. Image recognition uses...
  3. Doc 3 (score=1.2928): Natural language processing extracts meaning from text....
  4. Doc 1 (score=0.0000): Machine learning algorithms learn patterns from data. L...
  5. Doc 5 (score=0.0000): Reinforcement learning agents learn through rewards. Le...

BM25 is the default retrieval algorithm in Elasticsearch, OpenSearch, Apache Solr, and many other production search systems. For most English retrieval tasks, the default parameters ( $k_1 = 1.5$ , $b = 0.75$ ) work well without tuning. The next chapter covers BM25 in full depth, including its probabilistic derivation and the systematic approach to tuning $k_1$ and $b$ for specific corpora.