Dense Retrieval: Semantic Search & Bi-Encoder Implementation › From Lexical to Semantic Matching

Recall from Part II that BM25 retrieves documents by computing term-frequency statistics: documents score higher when they contain more query terms, with diminishing returns for repeated terms and penalties for common words. This approach works remarkably well for navigational queries, entity lookups, and any search where the vocabulary of the query and the relevant documents naturally aligns. The design is elegant in its simplicity: store an inverted index mapping each term to the documents that contain it, and at query time, look up each query term and aggregate the scores.

The vocabulary mismatch problem becomes acute in several important scenarios that arise constantly in real applications. Consider searching a medical knowledge base with the query "heart attack symptoms." A relevant document might discuss "myocardial infarction warning signs" without ever using the words "heart" or "attack." BM25 would score this document at zero because it shares no terms with the query. Yet a physician would immediately recognize these as describing the same condition. The mismatch here is between everyday language and technical terminology, and it is systematic: patients speak in plain language while medical literature uses precise clinical terms.

The vocabulary gap manifests in many forms, and each form defeats lexical retrieval in a different way:

• Synonyms and paraphrases: "car" vs "automobile," "purchase" vs "buy," "use" vs "utilize." Natural language has extraordinary breadth of expression for any given concept.
• Technical versus casual language: "myocardial infarction" vs "heart attack," "hypertension" vs "high blood pressure." Users often prefer informal phrasing while documents use formal or specialized terms.
• Abbreviations and expansions: "ML" vs "machine learning," "NLP" vs "natural language processing." Abbreviations are common in casual queries but documents may spell things out.
• Conceptual similarity without lexical overlap: "renewable energy policy" might relate to documents about "solar panel subsidies" or "wind farm regulation," even though these share no query terms.
• Different grammatical forms and derivations: "optimize" vs "optimization," "predict" vs "prediction." Stemming helps partially, but doesn't solve cross-form semantic relationships.

Dense retrieval sidesteps vocabulary mismatch entirely by operating in a different representational space. Instead of asking "which words match?", it asks "which meanings match?" By encoding both queries and documents into a continuous vector space where semantically similar texts cluster together, dense retrieval can find relevant documents regardless of the specific words they use.

The key insight is that transformer models, pre-trained on enormous amounts of text, develop internal representations that capture meaning at a level far beyond surface form. The model has seen "myocardial infarction" and "heart attack" used in similar contexts thousands of times, so their representations end up in similar regions of the embedding space. Dense retrieval uses this learned semantic geometry. When we ask the model to encode a query, it produces a vector that points in a direction in semantic space that represents the query's meaning. Document vectors that point in similar directions represent documents about related concepts.

This is a fundamentally different philosophy from lexical retrieval. In lexical retrieval, the representation is determined entirely by the words: the same words always produce the same representation, regardless of context. In dense retrieval, the representation is learned from data and captures distributional semantic relationships. Words that appear in similar contexts across millions of documents end up with similar representations, creating an implicit semantic knowledge graph embedded in the geometry of the vector space.