Reranking: Cross-Encoders for Precise Information Retrieval › The Reranking Procedure

Reranking is not a replacement for retrieval but a refinement layer. Understanding the full pipeline, from query to final ranked list, requires understanding each stage's purpose, its failure modes, and the parameters that govern its behavior. The complete pipeline follows a retrieve-then-rerank pattern, and each stage contributes a distinct kind of intelligence to the final result.

The first stage is designed for speed and recall. It casts a wide net, accepting some irrelevant documents in exchange for the assurance that relevant documents are unlikely to be missed. The second stage is designed for precision. It carefully reads the shortlist produced by the first stage, identifies the documents that satisfy the query, and moves them to the top. Neither stage is sufficient alone: retrieval without reranking produces results that are topically relevant but often imprecise; reranking without retrieval would be prohibitively slow at corpus scale.

The complete pipeline:

1. Initial Retrieval: Use fast methods (dense retrieval with bi-encoders, BM25 from BM25, or hybrid search from Hybrid Search) to fetch a candidate set of $k$ documents. This stage prioritizes recall over precision; you want to ensure the relevant documents are somewhere in this set, even if ranked poorly. The goal is complete coverage: you are willing to accept some irrelevant documents in the initial set to avoid missing any relevant ones. This is often called the "recall-oriented" stage of the pipeline.

2. Reranking: Feed each query-candidate pair through the cross-encoder to obtain precise relevance scores. Sort by these scores to produce a reranked list. This stage prioritizes precision: you want to identify exactly which of the candidates best answers the query, even if this process is computationally expensive. The cross-encoder acts as a careful judge, examining each candidate in detail to determine its true relevance.

3. Truncation: Return the top-$n$ documents from the reranked list (where $n \ll k$) to the downstream generation system or user. Typically, you might retrieve $k=100$ documents but only return the top $n=5$ or $n=10$. Because users rarely examine more than a handful of results, the top positions need the highest possible quality.

The parameter $k$ represents your reranking budget: how many candidates you can afford to score with the expensive cross-encoder. This is typically determined by latency constraints. If your cross-encoder takes 10ms per document and your latency budget is 100ms, you might set $k=10$. This budget is often one of the most necessary tuning parameters in production systems, requiring careful balancing between accuracy (higher $k$ gives the reranker more options) and user experience (lower $k$ ensures faster response times). In practice, $k$ is often determined empirically through latency testing and quality evaluation, with typical values ranging from 10 to 1000 depending on the computational resources available and the latency requirements of the application.

The key insight here is that the value of reranking depends critically on the quality of the initial retrieval. If the first-stage retriever fails to include the relevant document in its top $k$ results, no amount of reranking can recover it. This is why we speak of retrieve-then-rerank as a recall-then-precision strategy. You must ensure that the first stage achieves high recall, then rely on the second stage to improve precision within the recalled set. Measuring recall@k for the initial retriever on a validation set is therefore an needed diagnostic step before deploying any reranking system.

In large-scale systems, you might employ multiple reranking stages of increasing computational cost. Rather than applying the most expensive model to all candidates immediately, you use a sequence of progressively more sophisticated and costly models, each filtering the candidate set for the next stage. This creates a funnel where cheap models eliminate obviously irrelevant documents, leaving expensive models to focus computational effort only on the most promising candidates.

Think of cascade reranking as a talent competition with multiple rounds. In the first round, thousands of candidates perform a brief audition, and only the most promising advance. In the second round, those finalists perform a full set, and a smaller group moves to the finals. In the final round, a handful of top performers receive the most intensive scrutiny. Each round's judges spend proportionally more time per candidate, letting increasingly fine-grained discrimination. The key is that you never waste your most expensive judges on candidates who would clearly be eliminated in the first round.

For example, you might begin with a bi-encoder to retrieve one thousand documents, then apply a lightweight cross-encoder to select the best one hundred, followed by a heavy cross-encoder to identify the top ten, and finally employ a large language model as a judge to select the single best document or generate an answer based on the top three. Each stage in this cascade reduces the candidate set while increasing computational investment per document, letting you to apply your most expensive computational resources only where they matter most. This approach recognizes that most retrieved documents are clearly irrelevant and do not merit deep analysis, while a small subset requires careful scrutiny to distinguish subtle differences in relevance.

The engineering of cascade systems involves careful measurement at each stage. You need to know the precision-recall tradeoff of each filter, because overly aggressive filtering at an early stage can eliminate relevant documents that later stages can never recover. A common strategy is to run the full system offline on a validation set with varied cascade thresholds, then select thresholds that maintain a recall of at least 95% after each filtering stage.