Hybrid Search: BM25 and Dense Retrieval Combined › Dense Retrieval: The Semantic Half

Dense retrieval is a fundamentally different theory of relevance than keyword matching. Rather than asking which documents contain the same tokens as the query, dense retrieval asks which documents occupy the same region of a high-dimensional semantic space as the query. This move from token space to semantic space is what lets dense retrieval to find relevant documents that share no words with the query. But it also means the system depends entirely on the quality of the embedding model and can fail in predictable ways when that model's training distribution does not include the query's vocabulary.

As covered in depth in the Dense Retrieval and Embedding Models chapters, dense retrieval is both queries and documents as dense vectors in a shared embedding space, then retrieves documents whose vectors are nearest to the query vector. The key insight motivating this approach is that meaning can be encoded as position in a high-dimensional space: two pieces of text that mean similar things should occupy nearby positions, regardless of whether they share any words.

Think of dense retrieval as operating in a space where every possible meaning corresponds to a direction. Texts about "cardiac events" and texts about "heart attacks" both point in roughly the same direction in that space, because the model has learned from context that these phrases appear in the same semantic environments. Two documents that both discuss database indexing strategies also point in similar directions, even if one uses "B-tree" and the other uses "balanced tree search structure." The embedding model has captured a large web of these semantic associations and compressed it into the geometry of the embedding space.

This is fundamentally different from what BM25 does. BM25 operates on the surface form of text, counting occurrences of the literal tokens that appear in a query. Dense retrieval operates on a learned representation of meaning, where the model has internalized patterns from millions or billions of training examples and uses those patterns to place semantically similar texts near each other in the embedding space. The two approaches are looking at entirely different aspects of the text-retrieval relationship, which is exactly why combining them is so powerful.

A bi-encoder model encodes the query $q$ and each document $d$ independently into fixed-length vector representations:

$$\mathbf{q} = \text{Encoder}_Q(q), \quad \mathbf{d} = \text{Encoder}_D(d)$$

where:

• $\mathbf{q} \in \mathbb{R}^m$: the dense query embedding, a vector of $m$ real-valued dimensions
• $\mathbf{d} \in \mathbb{R}^m$: the dense document embedding in the same $m$-dimensional space
• $\text{Encoder}_Q$ and $\text{Encoder}_D$: transformer-based encoders (often the same shared model) that map text to vectors

The term "bi-encoder" refers to the architecture's property of encoding the query and the document through the same (or similarly structured) encoder independently, as opposed to a cross-encoder, which concatenates the query and document before encoding them jointly. The independence of encoding is necessary for practical retrieval: it means document embeddings can be computed once, stored offline, and then queried at inference time using only a single query encoding. This is what makes dense retrieval feasible at scale.

Retrieval ranks documents by cosine similarity between the query and document embeddings:

$$\text{score}(q, d) = \frac{\mathbf{q} \cdot \mathbf{d}}{|\mathbf{q}||\mathbf{d}|}$$

where:

• $\mathbf{q} \cdot \mathbf{d}$: the dot product of the query and document embedding vectors, summing the products of corresponding dimensions
• $|\mathbf{q}|$ and $|\mathbf{d}|$: the Euclidean norms (lengths) of the query and document vectors
• The result is bounded in $[-1, 1]$, with 1 meaning the vectors point in the same direction (maximum similarity) and 0 meaning they are orthogonal (no similarity)

The geometric interpretation of cosine similarity is worth dwelling on. Two vectors with cosine similarity near 1 point in approximately the same direction in the embedding space, regardless of how long each vector is. This direction-focus is what makes cosine similarity appropriate for comparing embeddings: the direction encodes the meaning, while the magnitude can vary with factors like token count or encoding normalization.

In practice, most bi-encoder systems normalize embeddings to unit length during encoding, which makes $|\mathbf{q}| = |\mathbf{d}| = 1$ and reduces cosine similarity to a simple dot product: $\text{score}(q, d) = \mathbf{q} \cdot \mathbf{d}$. This normalization has a practical benefit beyond mathematical simplicity: it means retrieval becomes a matrix-vector multiplication, which can be batched and parallelized efficiently on modern hardware. When the entire corpus embedding matrix is pre-normalized, a single matrix-vector product gives you cosine similarity scores for all documents simultaneously.

Dense retrieval has a clear strength and a clear weakness relative to BM25:

• Semantic generalization. If the model was trained on pairs where "heart attack" and "myocardial infarction" appear in semantically equivalent contexts, it will embed them near each other, and a query for one will retrieve documents about the other.
• Lexical precision. Dense models can miss exact matches for rare proper nouns, product codes, or technical identifiers that appear infrequently in training data. The model has not learned a reliable representation for "AB-1042-X" because it has almost never seen it. BM25, by contrast, will find every document containing that exact string regardless of training data.

This complementary failure pattern is the foundational motivation for hybrid search. It is not that one system is better than the other in some absolute sense. It is that they are better in different regimes, and those regimes can be exploited jointly.