BM25 › BM25

BM25 (Best Matching 25) is the algorithm that powers Elasticsearch and Solr, both of which use Lucene. It's been the standard in information retrieval since the 1990s and remains widely used today, even as neural approaches have emerged. Understanding BM25 gives you insight into what makes search work, and why simple term counting falls apart at scale.

Before neural language models rewrote the playbook, BM25 was the gold standard for text retrieval. Ask a search expert from 2010 what algorithm they trusted, and BM25 would come up within seconds. The remarkable thing is that this hasn't changed much even now. Production search systems at companies including Google and Microsoft still use BM25; Amazon does too. Sometimes it is the primary ranker, but more often it is the first stage in a hybrid pipeline that combines lexical matching with dense semantic retrieval. Understanding BM25 is not optional knowledge for anyone serious about search.

In this chapter, you'll learn where BM25 comes from (hint: probability theory), why it consistently outperforms TF-IDF, and how to implement it from scratch. We'll also explore its variants and see empirically how it stacks up against simpler approaches. Along the way, we'll cover the intuition behind each design decision so that the formula doesn't feel like a magic black box.