Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BM25 › Field-Weighted BM25

Michael BrenndoerferUpdated March 21, 20265 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 14BM25 › Field-Weighted BM25

Reading section 8 of 14 · Field-Weighted BM25 · 5 minSection 8 of 14 · 5 min

Section 8 of 14 · 5 min read

BM25 › Field-Weighted BM25One section · Sections in this chapter8 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Real documents often have structure: titles, abstracts, body text, metadata. A match in the title should probably count more than a match buried in the body. Field-weighted BM25 (sometimes called BM25F) addresses this.

BM25F (Field-Weighted BM25)

An extension of BM25 that handles structured documents by computing a weighted combination of term frequencies across different fields, with each field having its own boost weight and length normalization. This allows title matches to be weighted more heavily than body text matches.

The key idea is to compute a "virtual" term frequency that combines contributions from all fields:

\tilde{f}(t, D) = \sum_{c \in \text{fields}} w_c \cdot \frac{f(t, D_c)}{1 - b_c + b_c \cdot \frac{|D_c|}{\text{avgdl}_c}}

where:

$\tilde{f}(t, D)$ : virtual (combined) term frequency for term $t$ in document $D$
$c$ : a field in the document (e.g., title, body, abstract)
$w_c$ : boost weight for field $c$ (higher values give more importance to matches in that field)
$f(t, D_c)$ : frequency of term $t$ in field $c$ of document $D$
$b_c$ : length normalization parameter specific to field $c$
$|D_c|$ : length of field $c$ in document $D$
$\text{avgdl}_c$ : average length of field $c$ across all documents

This virtual term frequency then plugs into a simplified BM25 formula (note that length normalization is already applied per-field in $\tilde{f}$ ):

\text{BM25F}(D, Q) = \sum_{t \in Q} \text{IDF}(t) \cdot \frac{\tilde{f}(t, D) \cdot (k_1 + 1)}{\tilde{f}(t, D) + k_1}

where:

$D$ : the document being scored
$Q$ : the query (set of query terms)
$t$ : a term in the query
$\text{IDF}(t)$ : inverse document frequency of term $t$
$\tilde{f}(t, D)$ : virtual term frequency (computed from the formula above)
$k_1$ : saturation parameter (typically 1.2 to 2.0)

Notice that the length normalization term $(1 - b + b \cdot |D|/\text{avgdl})$ is absent from the denominator here because it's already incorporated into the virtual term frequency $\tilde{f}$ on a per-field basis. Each field has its own average length, so comparisons are made within fields rather than across the whole document.

An important subtlety: because $\tilde{f}$ is a linear combination of per-field term frequencies, all field lengths are compared against their own field averages. A title match against the average title length, and a body match against the average body length. This is much more principled than simply concatenating fields and doing standard BM25 on the concatenated text.

Let's implement a simple version:

In[25]:

Code

class BM25FIndex:
    def __init__(self, field_weights, k1=1.2, b=0.75):
        self.field_weights = field_weights  # {'title': 3.0, 'body': 1.0}
        self.k1 = k1
        self.b = b
        self.documents = []
        self.field_lengths = defaultdict(list)  # field -> [lengths]
        self.avgdl = {}  # field -> average length
        self.doc_freq = defaultdict(int)
        self.N = 0

    def tokenize(self, text):
        return re.findall(r"\b[a-z]+\b", text.lower())

    def index(self, documents):
        """Index documents with multiple fields.

        Each document should be a dict: {'title': '...', 'body': '...'}
        """
        self.documents = documents
        self.N = len(documents)

        # Compute document frequencies and field lengths
        all_terms = set()
        for doc in documents:
            doc_terms = set()
            for field in self.field_weights:
                tokens = self.tokenize(doc.get(field, ""))
                self.field_lengths[field].append(len(tokens))
                doc_terms.update(tokens)

            for term in doc_terms:
                self.doc_freq[term] += 1

        # Compute average lengths per field
        for field, lengths in self.field_lengths.items():
            self.avgdl[field] = sum(lengths) / len(lengths) if lengths else 1

    def compute_virtual_tf(self, doc_id, term):
        """Compute weighted, normalized term frequency across fields."""
        doc = self.documents[doc_id]
        virtual_tf = 0

        for field, weight in self.field_weights.items():
            tokens = self.tokenize(doc.get(field, ""))
            tf = tokens.count(term)
            if tf > 0:
                field_len = len(tokens)
                avgdl_field = self.avgdl[field]
                # Normalize by field length, then weight
                normalized_tf = tf / (
                    1 - self.b + self.b * (field_len / avgdl_field)
                )
                virtual_tf += weight * normalized_tf

        return virtual_tf

    def search(self, query, top_k=5):
        query_terms = self.tokenize(query)

        results = []
        for doc_id in range(self.N):
            score = 0
            for term in query_terms:
                if self.doc_freq.get(term, 0) == 0:
                    continue

                vtf = self.compute_virtual_tf(doc_id, term)
                if vtf > 0:
                    idf = math.log(
                        (self.N - self.doc_freq[term] + 0.5)
                        / (self.doc_freq[term] + 0.5)
                    )
                    score += idf * (vtf * (self.k1 + 1)) / (vtf + self.k1)

            if score > 0:
                results.append((doc_id, score))

        results.sort(key=lambda x: x[1], reverse=True)
        return results[:top_k]

Let's test with structured documents:

In[26]:

Code

# Structured documents with title and body
structured_docs = [
    {
        "title": "Introduction to Python Programming",
        "body": "This guide covers the basics of writing code in a popular scripting language.",
    },
    {
        "title": "Advanced Java Development",
        "body": "Learn enterprise patterns and best practices for building robust Python applications.",
    },
    {
        "title": "Web Development Fundamentals",
        "body": "Build modern websites using Python frameworks like Django and Flask.",
    },
    {
        "title": "Python for Data Science",
        "body": "Use Python libraries like pandas and numpy for data analysis and machine learning.",
    },
]

# Index with title weighted 3x more than body
bm25f_index = BM25FIndex(field_weights={"title": 3.0, "body": 1.0})
bm25f_index.index(structured_docs)

# Search
bm25f_results = bm25f_index.search("python programming", top_k=4)

Out[27]:

Console

Query: "python programming"
Field weights: title=3.0, body=1.0

------------------------------------------------------------

The results demonstrate how field weights influence ranking. Document 1 ranks highest because both "Python" and "Programming" appear in its title, which carries 3x the weight of the body field. Document 2 mentions "Python" only in the body text despite having "Development" in the title, so it ranks lower. This shows how BM25F can prioritize matches in more important fields, which is exactly the behavior you want in a product search or document retrieval system where title relevance means more than body relevance.