BM25: the Search Algorithm Behind Elasticsearch

BM25 (Best Matching 25) is the algorithm that powers Elasticsearch and Solr, both of which use Lucene. It's been the standard in information retrieval since the 1990s and remains widely used today, even as neural approaches have emerged. Understanding BM25 gives you insight into what makes search work, and why simple term counting falls apart at scale.

Before neural language models rewrote the playbook, BM25 was the gold standard for text retrieval. Ask a search expert from 2010 what algorithm they trusted, and BM25 would come up within seconds. The remarkable thing is that this hasn't changed much even now. Production search systems at companies including Google and Microsoft still use BM25; Amazon does too. Sometimes it is the primary ranker, but more often it is the first stage in a hybrid pipeline that combines lexical matching with dense semantic retrieval. Understanding BM25 is not optional knowledge for anyone serious about search.

In this chapter, you'll learn where BM25 comes from (hint: probability theory), why it consistently outperforms TF-IDF, and how to implement it from scratch. We'll also explore its variants and see empirically how it stacks up against simpler approaches. Along the way, we'll cover the intuition behind each design decision so that the formula doesn't feel like a magic black box.

Before diving into BM25, it helps to understand what problem it's solving. The obvious approach to search is to count how many times a query term appears in each document and rank by that count. This is called term frequency (TF) ranking, and it fails in predictable ways.

Consider a collection of news articles and the query "president." A full-length 10,000-word biography of a president will mention the word hundreds of times. A tight 200-word breaking news story might mention it twice. Raw term frequency ranking would always favor the longer document, even when the short article is more relevant to your query. Length bias is one of the most fundamental failure modes of naive retrieval.

The second failure mode is treating all terms equally. If your query is "the president of the United States," the word "the" appears in essentially every document. Counting "the" provides no discriminative signal. You want to reward documents for containing informative, specific terms, not common filler words.

The third failure mode is not recognizing diminishing returns. Once a document has established that it's about machine learning by mentioning the term ten times, the eleventh mention adds very little new information. Treating each occurrence as equally valuable leads to systems that can be gamed by keyword repetition.

BM25 solves all three of these problems with a single, elegant formula. It normalizes for document length, weights terms by how informative they are, and models diminishing returns from repeated occurrences. Let's see how.

BM25 didn't emerge from nowhere. It's the result of decades of research in probabilistic information retrieval, a field that asks: given a query, what's the probability that a document is relevant?

The story begins with the Binary Independence Model (BIM), developed in the 1970s. BIM makes two key assumptions:

1. Binary: Each term either appears in a document or it doesn't (we ignore term frequency)
2. Independence: Terms occur independently of each other given the relevance of a document

Under BIM, we want to rank documents by $P(R|D,Q)$, the probability that document $D$ is relevant given query $Q$. Using Bayes' theorem and the independence assumption, this leads to ranking by:

where:

• $t$: a term in both the query and document
• $Q \cap D$: the set of terms appearing in both query $Q$ and document $D$
• $P(t|R)$: probability of term $t$ appearing in a relevant document
• $P(t|\bar{R})$: probability of term $t$ appearing in a non-relevant document
• $R$: the event that a document is relevant
• $\bar{R}$: the event that a document is not relevant

The expression inside the logarithm is the odds ratio: it measures how much more likely a term is to appear in relevant documents compared to non-relevant ones. A term that's common in relevant documents but rare in non-relevant ones will have a high odds ratio, contributing positively to the score. A term equally common in both will have an odds ratio near 1, contributing little (since $\log(1) = 0$). We sum these contributions over all terms that appear in both the query and the document.

The difficulty is that we rarely have relevance judgments to estimate these probabilities. Robertson and Spärck Jones showed that with some simplifying assumptions, this reduces to something familiar: inverse document frequency.

The key simplifying assumption is to treat $P(t|R)$ as constant across terms (say 0.5, meaning any relevant document is equally likely to contain any given term), and to estimate $P(t|\bar{R})$ from the empirical document frequency in the collection. With these substitutions and a little algebra, the log-odds expression becomes proportional to the familiar IDF formula. The probabilistic model gives us a principled justification for something that might otherwise seem like a heuristic.

The Binary Independence Model ignores term frequency entirely, which is clearly a limitation. If a document mentions "machine learning" ten times, that should matter more than mentioning it once.

Robertson and Walker addressed this in the 1990s through a series of papers that culminated in BM25. The key insight was to incorporate term frequency in a principled way while maintaining the probabilistic foundation.

The derivation involves the 2-Poisson model, which assumes that term frequencies in relevant and non-relevant documents follow different Poisson distributions. Relevant documents are assumed to have a mixture of two Poisson processes: one for "elite" documents (those about the topic) and one for non-elite documents that happen to contain the term. Working through the math, this mixture model leads to a term frequency weighting that saturates. Additional occurrences provide diminishing returns, which is exactly the behavior we want.

The name "BM25" refers to the 25th model in a series of experiments conducted at the Robertson/Sparck Jones research group. BM11 and BM15 were earlier attempts; BM25 proved to be the most effective formulation. The number has no other special significance.

The full BM25 formula looks intimidating at first glance, but every piece exists for a reason. Rather than presenting it all at once, let's build it up from the three core problems any good retrieval function must solve:

1. Which terms matter? Not all words are equally informative. "The" appears everywhere; "quantum" is rare and specific.
2. How much does repetition help? If a document mentions your query term once, that's good. But is mentioning it 100 times really 100 times better?
3. How do we handle document length? A 10,000-word document naturally contains more vocabulary than a 100-word document. Should we penalize it for that?

BM25 addresses each of these problems with a specific component. Let's examine them one by one, then see how they combine into the complete formula.

Consider searching for "the cat sat on the mat." The word "the" appears in virtually every English document. Finding it tells you almost nothing. But "mat"? That's much more specific.

The Inverse Document Frequency (IDF) quantifies this intuition. A term that appears in many documents gets a low IDF; a rare term gets a high IDF. BM25 uses a specific IDF formulation derived from its probabilistic foundations:

where:

• $N$: total number of documents in the collection
• $n(t)$: number of documents containing term $t$
• The $+0.5$ terms provide smoothing to avoid division by zero and extreme values

This formulation differs from the standard $\log(N/n(t))$ IDF you might have seen in TF-IDF. The $N - n(t)$ in the numerator comes directly from the probabilistic derivation, representing the number of documents that don't contain the term. Intuitively, the formula computes a log-odds ratio: the numerator approximates the probability that a relevant document doesn't contain term $t$, while the denominator approximates the probability that a non-relevant document does contain it.

For very common terms appearing in more than half the documents, this IDF becomes negative. The formula effectively penalizes ubiquitous terms rather than just ignoring them. If a term appears in 80% of documents, matching on it might hurt your score slightly, pushing you to focus on more discriminative terms. Standard TF-IDF simply assigns a low positive weight to common terms; BM25's probabilistic derivation says such terms should actively count against a match.

Let's visualize how IDF varies with document frequency:

The plot reveals an important difference: standard IDF is always positive, merely approaching zero for common terms. BM25's IDF crosses zero at exactly 50% document frequency and becomes increasingly negative beyond that. This means stopwords like "the" or "a" that appear in nearly every document will have strongly negative IDF, effectively penalizing documents that match only on these terms. In practice, this is why BM25 implementations often include a floor at zero or simply filter out negative-IDF terms before scoring.

If a document mentions "machine learning" once, that's evidence of relevance. What about ten times? A hundred times?

Raw term frequency treats each occurrence equally: ten mentions = ten times the score. But this creates problems. A document that repeats a term obsessively (keyword stuffing) would dominate results. More fundamentally, there's a diminishing returns effect: the first mention tells you the document is about the topic; additional mentions provide less and less new information. The gap in relevance between a document that mentions a term once and a document that mentions it twice is much larger than the gap between 20 and 21 mentions.

BM25 captures this with a saturation function:

where:

• $f(t, D)$: frequency of term $t$ in document $D$
• $k_1$: saturation parameter controlling how quickly the function approaches its asymptote

The full formula includes length normalization in the denominator, which we'll add shortly.

To understand why this formula creates saturation, consider the mathematical behavior. When $f(t, D) = 0$ (term doesn't appear), the formula yields 0. When $f(t, D) = 1$ (term appears once), we get $(1 \cdot (k_1 + 1))/(1 + k_1) = 1$. As $f(t, D)$ grows very large, both the numerator and denominator grow at the same rate, so the ratio approaches $(k_1 + 1)$, an asymptote that cannot be exceeded no matter how many times the term appears.

You can verify this limit directly: as $f \to \infty$,

So the maximum possible contribution of any single query term, no matter how often it appears, is bounded by $k_1 + 1$ times the IDF weight. This cap makes BM25 resistant to keyword stuffing in a way that TF-IDF is not.

The intuition is this: the first occurrence of a term is strong evidence of relevance. Each additional occurrence provides weaker and weaker evidence, until eventually more occurrences barely matter at all.

The parameter $k_1$ controls how quickly you approach this ceiling:

• $k_1 = 0$: Instant saturation. Any non-zero term frequency gives the same score (like the Binary Independence Model)
• $k_1 \to \infty$: No saturation. Score is linear in term frequency (like raw TF-IDF)
• $k_1 \approx 1.2$ to $2.0$: The empirically validated sweet spot for most collections

Let's visualize this saturation effect:

The visualization reveals the saturation behavior clearly. With $k_1 = 0.5$, the curve flattens quickly: by term frequency 5, you've captured most of the possible score. With $k_1 = 5.0$, the curve is nearly linear over this range, meaning repetition continues to help. The default value of $k_1 = 1.2$ provides a principled middle ground: the first few occurrences matter a lot, but the score doesn't keep climbing indefinitely.

Here's a subtle but important issue. Imagine two documents about machine learning:

• Document A: A focused 200-word abstract that mentions "neural network" twice
• Document B: An extensive 10,000-word survey that mentions "neural network" twenty times

Which is more relevant to a query about neural networks? Raw term frequency would strongly favor Document B. But that's not necessarily right. Document B mentions the term more often simply because it's longer. The density of the term, its frequency relative to document length, might be a better signal.

But even this isn't the full story. Sometimes a long document has more relevant content: a detailed guide to neural networks should rank above a tweet that mentions the phrase once. The right approach is somewhere between ignoring length entirely and fully normalizing for it. BM25 lets you tune this tradeoff explicitly.

BM25 addresses this through length normalization, which adjusts term frequency based on how a document's length compares to the collection average:

where:

• $b$: length normalization parameter (typically between 0 and 1)
• $|D|$: length of document $D$ (number of terms)
• $\text{avgdl}$: average document length across the collection

This expression appears in the denominator of the BM25 formula, multiplied by $k_1$. Let's trace through what happens for different document lengths:

• Average-length document ($|D| = \text{avgdl}$): The ratio $|D|/\text{avgdl} = 1$, so the expression becomes $1 - b + b \cdot 1 = 1$. No adjustment.
• Longer document ($|D| > \text{avgdl}$): The ratio exceeds 1, so the expression is greater than 1. This increases the denominator, reducing the score.
• Shorter document ($|D| < \text{avgdl}$): The ratio is less than 1, so the expression is less than 1. This decreases the denominator, boosting the score.

The parameter $b$ controls how aggressively we normalize:

• $b = 0$: No length normalization. The expression always equals 1, and document length is ignored entirely.
• $b = 1$: Full length normalization. A document twice the average length has its term frequencies effectively halved.
• $b \approx 0.75$: The typical default. This provides moderate normalization that accounts for length without removing its effect.

The intuition behind $b = 0.75$ is that document length is partially informative. Some of the extra length in a long document reflects relevant content; some of it is just verbosity. The 0.75 setting is calibrated to discount verbosity while retaining some of the signal from long, content-rich documents.

Let's see how length normalization affects scoring:

The visualization confirms our intuition. When $b = 0$, document length doesn't matter at all: all curves are flat. With $b = 1$, a document that's three times average length gets significantly penalized. The default $b = 0.75$ provides a reasonable middle ground, acknowledging that longer documents might naturally contain more occurrences without completely ignoring the length difference.

Now we can assemble the complete BM25 formula. For each query term, we multiply three components:

1. IDF: How informative is this term?
2. Saturated TF: How often does it appear (with diminishing returns)?
3. Length adjustment: Normalized by document length

The final formula sums these contributions across all query terms:

where:

• $D$: the document being scored
• $Q$: the query (set of query terms)
• $t$: a term in the query
• $\text{IDF}(t)$: inverse document frequency of term $t$
• $f(t, D)$: frequency of term $t$ in document $D$
• $k_1$: saturation parameter (typically 1.2 to 2.0)
• $b$: length normalization parameter (typically 0.75)
• $|D|$: length of document $D$ (number of terms)
• $\text{avgdl}$: average document length in the collection

Each component addresses one of our original problems. The IDF weights rare terms more heavily. The saturation function ensures diminishing returns from repetition. The length normalization factor in the denominator adjusts for document size. Together, they create a scoring function that has worked well across decades of information retrieval research.

This formula does not include query term frequency. The original BM25 paper includes a query-side term frequency component as well (sometimes called BM25 with query term weighting), but in practice most implementations drop it. For typical short web queries, query terms appear at most once, making the query TF component degenerate to a constant. For longer queries, you could re-add it, but the gain is usually marginal.

The parameters $k_1$ and $b$ interact in interesting ways. Let's visualize how different combinations affect the score for a document with a fixed term frequency:

The heatmap reveals how the parameters affect the score. Moving right (higher $k_1$) increases scores because term frequency contributes more. Moving up (higher $b$) decreases scores because our document is longer than average and gets penalized more heavily. The default parameters sit in a region that balances these effects, but the optimal choice depends on your specific collection and retrieval goals.

With the formula understood, let's trace through a concrete example to see how all the pieces fit together. We'll score three short documents against a simple query, computing each component step by step.

Our corpus consists of several documents:

• D1: "the cat sat on the mat" (6 words)
• D2: "the cat sat on the cat mat" (7 words)
• D3: "the dog ran in the park" (6 words)

And our query is: "cat mat"

Notice that D1 and D2 both contain our query terms, but D2 mentions "cat" twice. D3 contains neither term. Let's see how BM25 handles these differences.

Before scoring any document, we need collection-level statistics. These establish the baseline against which individual documents are compared:

We have 7 documents with an average length of about 6 words. "Cat" appears in 2 of the 7 documents, and "mat" appears in 2 as well. These statistics feed into our IDF and length normalization calculations.

Now let's compute the IDF for each query term. Remember, IDF measures how discriminative a term is. Terms appearing in fewer documents get higher IDF:

Both terms appear in 2 out of 7 documents. Their IDF values reflect how specific they are relative to this collection. With a larger corpus, terms appearing in only 2 out of thousands of documents would receive much higher IDF values. This small corpus example illustrates the mechanics, but the dynamics play out more clearly at scale.

Now we combine everything: IDF, saturated term frequency, and length normalization. For each query term in each document, we compute the contribution and sum them up:

The visualization makes the scoring dynamics clear:

• D2 wins despite being the longest document. Why? It contains "cat" twice, and the saturation effect hasn't fully kicked in at TF=2. The extra occurrence still provides meaningful additional score.
• D1 comes second with both query terms appearing once each. It's slightly shorter than average, which gives it a small boost.
• D3 scores zero because it contains neither query term. No amount of length normalization or parameter tuning can help a document that lacks the query terms entirely.

The negative scores might seem strange, but they're a consequence of our small corpus where both query terms are common relative to the collection size. In practice, with larger corpora and more discriminative query terms, you'd see positive scores for matching documents. What matters is the relative ranking, and BM25 correctly identifies D2 as most relevant.

Understanding the formula is one thing; implementing it efficiently is another. Let's build a complete BM25 retrieval system step by step, seeing how the mathematical concepts translate into code.

We'll use a larger corpus to make the results more interesting and realistic:

A BM25 index needs to store several pieces of information efficiently:

1. Document frequencies: For each term, how many documents contain it? (Needed for IDF)
2. Document lengths: How long is each document? (Needed for length normalization)
3. Average document length: The collection-wide baseline for length comparison
4. Inverted index: For each term, which documents contain it and with what frequency? (Needed for efficient retrieval)

The inverted index is the key data structure that makes search practical at scale. Without it, you'd need to scan every document for every query, which becomes prohibitive once you have millions of documents. With an inverted index, you can immediately look up which documents contain each query term and score only those candidates. This reduces the scoring workload from O(N) per query to O(postings), where postings is the number of documents containing any query term.

Let's create a class that builds and stores this index:

The indexing process makes a single pass through the corpus, computing all the statistics we need. One subtle point: we store each term's (doc_id, tf) pairs in the inverted index during indexing. This means we don't need to re-tokenize documents at query time.

Now we add methods to compute BM25 scores. The scoring function directly implements our formula, computing IDF, saturated term frequency, and length normalization for each query term:

Notice how the search method uses the inverted index to find candidate documents efficiently. Instead of scoring all documents, we only consider those containing at least one query term. This optimization matters for large corpora where most documents won't match the query at all. A query for "quantum entanglement" against a collection of 10 million news articles would match perhaps a few hundred documents; there's no need to score the other 9,999,700.

Let's test our BM25 implementation with several queries to see how it ranks documents:

Our BM25 implementation successfully retrieves relevant documents. Several patterns emerge from these results:

• Specific queries work well: "machine learning" finds the one document explicitly about ML and data science.
• Multiple matches boost scores: Documents matching both query terms (like "Python programming") score higher than those matching just one.
• IDF matters: Common terms like "programming" contribute less than rarer terms like "machine" or "enterprise."

The implementation is functionally complete, but production systems like Elasticsearch add many optimizations: compressed inverted indexes, skip lists for faster intersection, block-max optimizations that skip posting lists when their maximum score can't enter the top-k, and caching of IDF values across queries.

The original BM25 formula has some known issues that researchers have addressed with variants. Understanding these variants helps you make better choices when the default BM25 underperforms in specific situations.

One issue with BM25 is that very short documents can be over-penalized. If a document is much shorter than average, the length normalization term $1 - b + b \cdot \frac{|D|}{\text{avgdl}}$ becomes less than 1, which boosts the term frequency component. But this boost might not be enough to compensate for the document simply having fewer terms.

Consider a collection of academic abstracts with an average length of 200 words, but you've added some very short 20-word author bio entries. Those bio entries might contain a query term once, but because they're so short relative to the average, the length normalization boost might make them score oddly high. More seriously, the normalization approaches break down mathematically when documents are extremely short.

More fundamentally, there's a theoretical concern: Lv and Zhai (2011) showed that the original BM25 violates a property called the "term frequency normalization lower bound." This property says that a document containing a query term should always score higher than the same document without the query term. At extreme length differences, standard BM25 can violate this. The variants below fix this.

BM25L (BM25 with Lower bound) addresses this by modifying how term frequency is normalized. First, it computes a length-normalized term frequency:

where:

• $c(t, D)$: the length-normalized term frequency
• $f(t, D)$: raw frequency of term $t$ in document $D$
• $b$: length normalization parameter (typically 0.75)
• $|D|$: length of document $D$ (number of terms)
• $\text{avgdl}$: average document length across the collection

This formula essentially "inflates" the term frequency for short documents and "deflates" it for long documents. A document half the average length would have its term frequency doubled; a document twice the average length would have it halved.

Then BM25L adds a lower bound constant $\delta$ to this normalized value:

where:

• $D$: the document being scored
• $Q$: the query (set of query terms)
• $t$: a term in the query
• $\text{IDF}(t)$: inverse document frequency of term $t$
• $c(t, D)$: length-normalized term frequency (defined above)
• $\delta$: lower bound constant (typically 0.5)
• $k_1$: saturation parameter (same as in standard BM25)

The addition of $\delta$ to the normalized term frequency ensures that even a single occurrence of a term makes a nonzero contribution to the score, regardless of document length. This $\delta$ acts as a guarantee: the term frequency component can never fall below $\delta/(k_1 + \delta)$, which is strictly positive. This restores the lower bound property that standard BM25 can violate.

BM25+ takes a different approach by adding a constant to the entire term weight:

where:

• $D$: the document being scored
• $Q$: the query (set of query terms)
• $t$: a term in the query
• $\text{IDF}(t)$: inverse document frequency of term $t$
• $f(t, D)$: frequency of term $t$ in document $D$
• $k_1$: saturation parameter (typically 1.2 to 2.0)
• $b$: length normalization parameter (typically 0.75)
• $|D|$: length of document $D$ (number of terms)
• $\text{avgdl}$: average document length in the collection
• $\delta$: additive constant (typically 1.0)

The key insight is that $\delta$ is added inside the IDF multiplication. This guarantees that a document containing a query term always scores higher than one that doesn't, regardless of length normalization effects. Even if the term frequency component becomes very small due to aggressive length normalization, the $\delta$ ensures a minimum positive contribution. The IDF scaling means that this floor is higher for rare terms (where it matters more) than for common terms.

Let's compare these variants:

BM25+ consistently gives higher scores, especially for shorter documents where the additive $\delta$ term has more relative impact. BM25L provides a middle ground with its adjusted term frequency calculation. For most practical applications, the differences are small, and the defaults work well. The variants matter primarily when your collection has a mix of very short and very long documents, such as a mix of social media posts and full articles.

Real documents often have structure: titles, abstracts, body text, metadata. A match in the title should probably count more than a match buried in the body. Field-weighted BM25 (sometimes called BM25F) addresses this.

The key idea is to compute a "virtual" term frequency that combines contributions from all fields:

where:

• $\tilde{f}(t, D)$: virtual (combined) term frequency for term $t$ in document $D$
• $c$: a field in the document (e.g., title, body, abstract)
• $w_c$: boost weight for field $c$ (higher values give more importance to matches in that field)
• $f(t, D_c)$: frequency of term $t$ in field $c$ of document $D$
• $b_c$: length normalization parameter specific to field $c$
• $|D_c|$: length of field $c$ in document $D$
• $\text{avgdl}_c$: average length of field $c$ across all documents

This virtual term frequency then plugs into a simplified BM25 formula (note that length normalization is already applied per-field in $\tilde{f}$):

where:

• $D$: the document being scored
• $Q$: the query (set of query terms)
• $t$: a term in the query
• $\text{IDF}(t)$: inverse document frequency of term $t$
• $\tilde{f}(t, D)$: virtual term frequency (computed from the formula above)
• $k_1$: saturation parameter (typically 1.2 to 2.0)

Notice that the length normalization term $(1 - b + b \cdot |D|/\text{avgdl})$ is absent from the denominator here because it's already incorporated into the virtual term frequency $\tilde{f}$ on a per-field basis. Each field has its own average length, so comparisons are made within fields rather than across the whole document.

An important subtlety: because $\tilde{f}$ is a linear combination of per-field term frequencies, all field lengths are compared against their own field averages. A title match against the average title length, and a body match against the average body length. This is much more principled than simply concatenating fields and doing standard BM25 on the concatenated text.

Let's implement a simple version:

The results demonstrate how field weights influence ranking. Document 1 ranks highest because both "Python" and "Programming" appear in its title, which carries 3x the weight of the body field. Document 2 mentions "Python" only in the body text despite having "Development" in the title, so it ranks lower. This shows how BM25F can prioritize matches in more important fields, which is exactly the behavior you want in a product search or document retrieval system where title relevance means more than body relevance.

How much does BM25's sophistication matter? Let's compare it directly against TF-IDF on a retrieval task.

BM25 scores tend to have a wider dynamic range with clearer separation between relevant and non-relevant documents. TF-IDF cosine similarities are bounded between 0 and 1, which can make it harder to distinguish between moderately relevant documents. This separation is important in practice: when you're taking the top-10 results out of millions, you want the scoring function to clearly distinguish the winner from the also-rans.

While understanding BM25 internals is valuable, in practice you'll often use existing implementations. The rank_bm25 library provides a simple, efficient implementation:

The library returns results consistent with our from-scratch implementation, correctly identifying documents about neural networks and training. The rank_bm25 library handles tokenization and indexing, then performs scoring internally, making it convenient for rapid prototyping.

For production search systems, Elasticsearch and OpenSearch use BM25 as their default scoring algorithm. They also support BM25F for multi-field documents out of the box. Elasticsearch exposes the k1 and b parameters directly in the index mapping settings, letting you tune them without any code changes.

In Elasticsearch, BM25 is configured per-field in the index mapping. Here's how you'd set custom parameters:

The typical workflow for tuning BM25 in Elasticsearch starts with the defaults, then adjusts based on the characteristics of your collection. If your documents are very uniform in length (like product descriptions), you might lower $b$ toward 0.5. If your collection has a lot of spam or keyword stuffing, you might lower $k_1$ toward 1.0 to apply more aggressive saturation.

BM25 works well in practice, but it has fundamental limitations that become apparent as soon as you push beyond exact keyword matching.

The most significant limitation is vocabulary mismatch. BM25 only matches exact terms. If your query says "car" but the document says "automobile," BM25 won't find the connection. This is not a minor edge case: it's a fundamental restriction of lexical matching that affects every retrieval task. Users phrase queries in their own words, which may not match the words authors chose when writing documents. Synonyms and paraphrases create vocabulary gaps, as do abbreviations and spelling variants that BM25 cannot bridge.

Closely related is the lack of semantic understanding. "Bank of the river" and "bank account" would be treated identically by BM25: it sees the tokens, not the meaning. A query about financial institutions would retrieve documents about riverbanks if they happen to use the word "bank" heavily. This kind of lexical ambiguity is pervasive in natural language, and BM25 has no mechanism to resolve it.

The bag-of-words assumption also discards word order entirely. "Dog bites man" scores the same as "Man bites dog" for a query about dogs and men, despite having very different meanings. Phrase queries are a partial workaround (Elasticsearch supports them with match_phrase), but they require exact contiguous matches and don't generalize to paraphrases.

Finally, BM25 does not capture relationships between query terms. A query for "machine learning" is treated as two independent queries for "machine" and "learning." A document about industrial machinery that also happens to discuss learning processes might score well, even though it's irrelevant to the query's intent.

Despite these limitations, BM25 remains widely used because of several practical advantages:

• Speed: BM25 requires only sparse vector operations over the inverted index, making it extremely fast even on billion-document collections.
• Interpretability: You can explain exactly why a document ranked where it did. The score is a sum of per-term contributions, each traceable back to the formula.
• No training required: BM25 works out of the box on any text collection without labeled examples or GPU compute.
• Handling rare terms: Neural retrievers struggle with rare or unseen terms (out-of-vocabulary words). BM25 handles them naturally since it only requires the term to appear in the index.

Modern systems often combine BM25 with neural retrievers through hybrid search, getting the best of both worlds.

The typical hybrid pipeline works as follows: BM25 retrieves a first-stage set of candidates (say, the top 100 or 1000 documents), then a neural re-ranker like a cross-encoder re-scores those candidates and selects the final top-10. This is much cheaper than running a neural model over the entire collection, while still allowing semantic understanding in the final ranking stage. Elasticsearch and Vespa have native support for hybrid retrieval with BM25 plus dense vectors; Weaviate does as well.

Another emerging pattern is to use BM25 alongside dense retrievers in parallel, then fuse their rankings with a technique called Reciprocal Rank Fusion (RRF). Each system contributes its rankings independently, and the fusion step combines them without needing to calibrate the score scales. This sidesteps the difficult problem of making BM25 scores and cosine similarities comparable.

BM25 represents the high point of classical information retrieval. Its design embodies three carefully motivated ideas: terms should be weighted by how informative they are (IDF), additional occurrences provide diminishing returns (saturation), and longer documents deserve proportional scrutiny (length normalization). Each parameter in the formula was validated empirically across many test collections, which is why the defaults work well in practice.

The key takeaways from this chapter:

• Probabilistic foundation: BM25 derives from the Binary Independence Model and the 2-Poisson model, giving it a principled basis that TF-IDF lacks.
• Term frequency saturation: The $k_1$ parameter ensures diminishing returns from repeated terms, with typical values between 1.2 and 2.0.
• Length normalization: The $b$ parameter prevents long documents from unfairly dominating, with 0.75 as the empirically validated default.
• IDF weighting: Rare terms receive higher weight, and terms appearing in more than half the collection receive negative IDF in the BM25 formulation.
• Variants exist: BM25L and BM25+ fix edge cases with short documents, while BM25F handles structured documents with multiple fields.

The standard parameters ($k_1 = 1.2$, $b = 0.75$) work well across most collections, though tuning can help for specific domains. Decreasing $k_1$ suppresses keyword stuffing; decreasing $b$ helps when document length variation is meaningful. BM25+ or BM25L becomes worth considering when your collection has extreme length variation.

BM25's lasting influence is evident in its continued use in Elasticsearch and Solr, among other search engines. Even as neural retrievers have emerged, BM25 often is the first-stage retriever or as part of hybrid systems where exact lexical matching complements semantic understanding. Understanding BM25 gives you insight into what makes search work at a fundamental level, and why modern approaches needed to go beyond lexical matching to achieve true semantic understanding of language.

Understanding BM25's parameters is essential for tuning retrieval performance:

Tuning guidance:

• Start with defaults ($k_1 = 1.2$, $b = 0.75$): These work well for most text collections and are the defaults in Elasticsearch and Lucene.
• Increase $k_1$ (toward 2.0) when term frequency is highly informative, such as in longer documents where repeated mentions indicate relevance.
• Decrease $k_1$ (toward 0.5) for collections where a single mention is as informative as multiple mentions, such as short documents or when keyword stuffing is a concern.
• Decrease $b$ (toward 0.25) when document length variation is meaningful, for example, when longer documents contain more relevant information rather than just more filler.
• Increase $b$ (toward 1.0) when longer documents tend to match queries spuriously due to containing more vocabulary.
• Use BM25+ or BM25L when your collection has highly variable document lengths and you observe short, relevant documents being ranked too low.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about BM25 and information retrieval.