BM25 (1994) › The Problem: The Limitations of Traditional Ranking
History of Language AIAll chapters BM25 (1994) › The Problem: The Limitations of Traditional RankingOne section · Sections in this chapter2 / 7 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
TF-IDF multiplies a term's frequency in one document by a measure of how rare that term is across the collection. A term that occurs often in a document but rarely elsewhere receives a high weight. The basic formulation, however, does not saturate repeated occurrences or normalize document length in the way BM25 does.
One limitation was the linear treatment of term frequency: a document containing a query term ten times could receive twice the term-frequency contribution of one containing it five times. Relevance rarely grows at that rate. Once a document is clearly about machine learning, another ten mentions of the phrase add little evidence.
Document length created another bias. Longer documents have more opportunities to contain query terms, so a 50-page technical report might outrank a focused two-page summary even when the summary better answers the query.
TF-IDF also uses inverse document frequency as a heuristic rather than deriving the full score from a relevance model. That makes its behavior useful in practice but harder to connect to explicit assumptions about relevant and non-relevant documents.
Term weights also depend on collection statistics rather than an interpretation of the query. In "machine learning algorithms for natural language processing," TF-IDF can weight rare terms more heavily, but it does not recognize "machine learning" or "natural language processing" as phrases with meanings beyond their individual words.