Reranking: Cross-Encoders for Precise Information Retrieval › The Limitations of Bi-Encoders

To understand why reranking is necessary, you must first appreciate the constraints of bi-encoder retrieval. In bi-encoder architectures, which we covered in Dense Retrieval and Embedding Models, the query and document are encoded independently. The scoring function takes the form:

$$\text{score}(q, d) = \text{sim}(f_\theta(q), g_\phi(d))$$

where:

• $q$: the query text
• $d$: the document text
• $f_\theta$: the query encoder (a transformer network with parameters $\theta$)
• $g_\phi$: the document encoder (a transformer network with parameters $\phi$)
• $\text{sim}(\cdot, \cdot)$: similarity function (typically cosine similarity or dot product)

This factorization enables precomputation: you encode your document corpus once and index the vectors for fast nearest-neighbor search. This architectural choice makes bi-encoders remarkably efficient at scale, letting millions or billions of documents to be searched in milliseconds. The document vectors are computed offline, and at query time you only need to encode the query and perform a fast approximate nearest-neighbor lookup. This scalability is what makes bi-encoders needed in production systems.

However, independence comes at a cost. When you encode a document, you must produce a single vector representation without knowing what query it will be compared against. This creates an asymmetric compression problem: the document encoder must anticipate all possible questions you might ask and pack the answer into a fixed-dimensional vector. For complex documents containing multiple facts or fine-grained arguments, this is inherently lossy. The encoder essentially performs lossy compression of the document's semantic content, averaging over potentially relevant information in a way that preserves general topical similarity but often obliterates the specific details that determine relevance to a particular query. From an information-theoretic perspective, the document encoder must construct a sufficient statistic for all possible queries, which is impossible for documents containing many distinct facts and arguments.

The key insight is that the independence assumption is not a bug but a deliberate architectural choice that trades representation fidelity for precomputability. The bi-encoder cannot be more accurate without sacrificing the ability to precompute document vectors, because any interaction between query and document tokens would require knowing the query before encoding the document. This tradeoff is basic, not accidental.

Consider a query: "What temperature causes chocolate to bloom?" A bi-encoder might retrieve a document about chocolate manufacturing that mentions blooming, but it cannot distinguish whether the document states the specific temperature threshold or discusses blooming as a general phenomenon. The embedding similarity captures semantic relatedness but not factual sufficiency. The vector representation might encode that the document discusses "chocolate," "bloom," and "temperature" in close semantic proximity, but it cannot verify whether the specific causal relationship or numerical value requested by the query is present in the text. This distinction between topical relevance and factual sufficiency lies at the heart of the bi-encoder's limitations.

Think of the bi-encoder as a fingerprint scanner. It can tell you that two things share the same general shape and pattern, but it cannot tell you whether a specific detail, a particular scar, a unique ridge, is present. The cross-encoder, by contrast, is a forensic analyst who examines every feature of both prints side by side, looking for specific matches and mismatches. The analyst is slower, but the analyst's judgment is far more reliable for cases where the difference between relevant and irrelevant comes down to a single specific detail.

Beyond factual sufficiency, bi-encoders also struggle with negation and contrast. If a query asks "What are the risks of using aspirin?" and a document discusses the benefits of aspirin, the embedding similarity may still be high because both texts share the topic of aspirin. A cross-encoder can detect the conceptual mismatch between "risks" in the query and "benefits" in the document because it processes them together and can attend to the semantic polarity of each claim. Similarly, queries involving temporal qualifiers ("What changed after 2020?"), causal relationships ("Why did X cause Y?"), or conditional constraints ("How do you do X without Y?") all present challenges for bi-encoders that cross-encoders handle more naturally through joint reasoning over the full text.