Reranking: Cross-Encoders for Precise Information Retrieval › Reranking

After retrieving candidate documents using dense vectors or hybrid search, you face a basic limitation: embedding similarity is a coarse measure of relevance. As we discussed in Dense Retrieval, bi-encoders compress documents into fixed-length vectors to enable fast approximate search, but this compression discards fine-grained interactions between query and document tokens. A document might share vocabulary with your query yet answer a completely different question, or it might contain the answer buried in irrelevant context that the embedding averages away. This loss of fidelity creates an information bottleneck that limits the precision of first-stage retrieval systems. When you ask a specific question requiring a fine-grained answer, the bi-encoder's single-vector summary often lacks the discriminative power to distinguish between superficially similar documents with different informational content.

Think of bi-encoder retrieval as a librarian who quickly scans the titles and back covers of thousands of books to create a shortlist for you. The librarian is fast and generally right about which shelf to check, but they cannot read every page of every book. Reranking is the second step: you hand that shortlist to an expert who opens each book, reads the relevant passages, and tells you which one answers your question. The expert is slow, but once the librarian has narrowed the field to a manageable set, the expert's careful judgment turns an approximate result into a precise one.

Reranking solves this by introducing a second, more careful judgment phase. Instead of comparing precomputed vectors, a reranker processes the query and candidate document together through a deep cross-attention mechanism. This produces a precise relevance score. This two-stage architecture, retrieve-then-rerank, has become the standard pattern for high-accuracy information retrieval systems. You first cast a wide net with fast bi-encoders to recall potentially relevant documents, then apply a computationally expensive but accurate cross-encoder to the top candidates to rank them precisely. This architectural pattern recognizes an needed truth about information retrieval: efficiency and accuracy often demand different computational strategies. The initial retrieval stage optimizes for speed and recall. This keeps no relevant document is overlooked, while the reranking stage optimizes for precision. This keeps that the most relevant documents surface to the top.

The distinction matters in practice. Modern search engines, question-answering systems, and retrieval-augmented generation (RAG) pipelines all depend on this two-stage design. Without the first stage, the reranker would have to score millions of documents, which is computationally impossible at interactive latencies. Without the second stage, the system would return documents that look topically relevant but fail to answer the user's specific question. This chapter examines how the two stages work together and the engineering decisions required to balance their competing demands.

This chapter examines the cross-encoder architecture that powers modern reranking, the procedures for integrating reranking into retrieval pipelines, how to train rerankers effectively, and the necessary latency tradeoffs that determine when and how to deploy them. We will also explore intermediate architectures like ColBERT that occupy the efficiency-accuracy spectrum between pure bi-encoders and full cross-encoders. Understanding these elements enables you to balance accuracy against speed within a fixed computational budget. Along the way, we will pay close attention to the mathematical foundations that explain why each design choice works, and we will ground every concept in concrete examples that make the abstractions tangible.