BM25 › The Problem with Counting Words

Before diving into BM25, it helps to understand what problem it's solving. The obvious approach to search is to count how many times a query term appears in each document and rank by that count. This is called term frequency (TF) ranking, and it fails in predictable ways.

Consider a collection of news articles and the query "president." A full-length 10,000-word biography of a president will mention the word hundreds of times. A tight 200-word breaking news story might mention it twice. Raw term frequency ranking would always favor the longer document, even when the short article is more relevant to your query. Length bias is one of the most fundamental failure modes of naive retrieval.

The second failure mode is treating all terms equally. If your query is "the president of the United States," the word "the" appears in essentially every document. Counting "the" provides no discriminative signal. You want to reward documents for containing informative, specific terms, not common filler words.

The third failure mode is not recognizing diminishing returns. Once a document has established that it's about machine learning by mentioning the term ten times, the eleventh mention adds very little new information. Treating each occurrence as equally valuable leads to systems that can be gamed by keyword repetition.

BM25 solves all three of these problems with a single, elegant formula. It normalizes for document length, weights terms by how informative they are, and models diminishing returns from repeated occurrences. Let's see how.