Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BM25 › Using BM25 in Practice

Michael BrenndoerferUpdated March 21, 20262 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 14BM25 › Using BM25 in Practice

Reading section 10 of 14 · Using BM25 in Practice · 2 minSection 10 of 14 · 2 min

Section 10 of 14 · 2 min read

BM25 › Using BM25 in PracticeOne section · Sections in this chapter10 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While understanding BM25 internals is valuable, in practice you'll often use existing implementations. The rank_bm25 library provides a simple, efficient implementation:

In[33]:

Code

# Install if needed: uv pip install rank-bm25
from rank_bm25 import BM25Okapi

# Tokenize corpus
tokenized_corpus = [doc.lower().split() for doc in test_corpus]

# Create BM25 index
bm25 = BM25Okapi(tokenized_corpus)

# Search
query = "neural network training optimization"
tokenized_query = query.lower().split()
scores = bm25.get_scores(tokenized_query)

Out[34]:

Console

Query: "neural network training optimization"

Top 5 results using rank_bm25:
------------------------------------------------------------
1. [Score: 6.2540] Backpropagation computes gradients for neural network training
2. [Score: 2.1638] Supervised learning requires labeled training examples
3. [Score: 1.6298] Neural networks are inspired by biological neurons
4. [Score: 1.5328] Deep learning uses neural networks with many layers
5. [Score: 0.0000] Named entity recognition identifies people places and organizations

The library returns results consistent with our from-scratch implementation, correctly identifying documents about neural networks and training. The rank_bm25 library handles tokenization and indexing, then performs scoring internally, making it convenient for rapid prototyping.

For production search systems, Elasticsearch and OpenSearch use BM25 as their default scoring algorithm. They also support BM25F for multi-field documents out of the box. Elasticsearch exposes the k1 and b parameters directly in the index mapping settings, letting you tune them without any code changes.

BM25 in ElasticsearchLink Copied

In Elasticsearch, BM25 is configured per-field in the index mapping. Here's how you'd set custom parameters:

In[5]:

Code

# Elasticsearch index mapping with custom BM25 parameters
mapping = {
    "settings": {
        "index": {
            "similarity": {"custom_bm25": {"type": "BM25", "k1": 1.5, "b": 0.6}}
        }
    },
    "mappings": {
        "properties": {
            "title": {"type": "text", "similarity": "custom_bm25"},
            "body": {
                "type": "text",
                "similarity": "BM25",  # Default parameters
            },
        }
    },
}

The typical workflow for tuning BM25 in Elasticsearch starts with the defaults, then adjusts based on the characteristics of your collection. If your documents are very uniform in length (like product descriptions), you might lower $b$ toward 0.5. If your collection has a lot of spam or keyword stuffing, you might lower $k_1$ toward 1.0 to apply more aggressive saturation.