Reranking: Cross-Encoders for Precise Information Retrieval › Training Rerankers

Training a cross-encoder reranker requires supervised data consisting of query-document pairs labeled with relevance judgments. Unlike bi-encoders, which we trained using contrastive objectives in Contrastive Learning for Retrieval, cross-encoders can use standard classification or ranking losses because they process pairs jointly. This joint processing means that the model can learn directly from explicit relevance signals rather than needing to construct contrastive pairs that approximate the ranking task. The availability of labeled relevance judgments, typically obtained through human annotators or implicit feedback signals like clicks, allows cross-encoders to optimize directly for the probability of relevance or relative ranking preferences.

The choice of training objective has significant practical consequences. Binary classification losses train the model to output calibrated probabilities, which is useful when you want to apply a fixed threshold to determine whether any document is relevant. Pairwise ranking losses train the model to correctly order document pairs, which is more directly aligned with the ranking task but may produce poorly calibrated scores. Listwise losses optimize the entire ranked list simultaneously. This provides the strongest alignment with end-to-end ranking quality but requiring richer supervision signals. In practice, a combination of these objectives often works best, with pairwise or listwise losses used during primary training and calibration steps applied afterwards.

The simplest approach treats relevance as a binary classification problem. Given a dataset $\{(q_i, d_i, y_i)\}_{i=1}^N$ where $y_i \in \{0, 1\}$ indicates relevance, you minimize cross-entropy loss. For each training example, the loss measures the discrepancy between the predicted probability and the true binary label. The cross-entropy formulation naturally handles the probabilistic nature of relevance judgments and provides smooth gradients that enable efficient training via gradient descent:

$$\mathcal{L} = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log \sigma(s(q_i, d_i)) + (1 - y_i) \log(1 - \sigma(s(q_i, d_i))) \right]$$

where:

• $N$: number of training examples
• $q_i$: the $i$-th query in the training set
• $d_i$: the $i$-th document in the training set
• $y_i \in \{0, 1\}$: binary relevance label (1 if relevant, 0 if irrelevant)
• $s(q_i, d_i)$: predicted relevance score for the $i$-th pair
• $\sigma(\cdot)$: sigmoid function converting scores to probabilities
• $\mathcal{L}$: the cross-entropy loss measuring prediction error

When $y_i = 1$ (relevant), the loss reduces to $-\log \sigma(s(q_i, d_i))$, which is minimized when the model assigns a high score to the pair. When $y_i = 0$ (irrelevant), the loss becomes $-\log(1 - \sigma(s(q_i, d_i)))$, which is minimized when the model assigns a low score. The average over all training examples pushes the model to simultaneously score all relevant documents highly and all irrelevant documents lowly. The key property of this loss is that the gradient does not vanish until the model is extremely confident in the correct direction. This keeps training continues to make meaningful progress even when the model is already doing reasonably well.

This works well when you have explicit relevance labels, such as click-through data or human judgments on a binary scale (relevant/irrelevant). However, collecting reliable binary labels presents significant challenges. Human annotators often disagree on relevance boundaries, and click-through data can be noisy due to position bias (users are more likely to click top results regardless of relevance) and presentation bias (attractive titles may generate clicks even for irrelevant documents). Active learning strategies can help prioritize which documents to label, focusing annotation effort on the boundary cases where the model is most uncertain. Additionally, class imbalance is common in retrieval datasets: for most queries, the large majority of documents are irrelevant, so you may need to use weighted loss functions or careful sampling strategies to prevent the model from learning to always predict "irrelevant."

Often, you have relative judgments: for query $q$, document $d^+$ is more relevant than $d^-$. Rather than requiring absolute relevance labels, pairwise approaches only need to know the relative ordering between pairs. This is a more natural fit for how humans perceive relevance: it is often easier to say "Document A is better than Document B for this query" than to assign a specific numeric relevance score to each document in isolation.

The margin ranking loss encourages the model to score the positive higher than the negative by at least a margin $m$:

$$\mathcal{L} = \sum_{(q, d^+, d^-)} \max(0, m - s(q, d^+) + s(q, d^-))$$

where:

• $q$: the query being processed
• $d^+$: a document labeled as relevant (positive) to query $q$
• $d^-$: a document labeled as irrelevant (negative) to query $q$
• $s(q, d)$: relevance score assigned to query-document pair $(q, d)$
• $m$: margin hyperparameter (typically set to 1.0) specifying the minimum desired score difference
• $\mathcal{L}$: the margin ranking loss penalizing violations of the ranking constraint

This is analogous to the hinge loss used in support vector machines. If $s(q, d^+) \geq s(q, d^-) + m$, the loss is zero; otherwise, it increases linearly with the violation. The margin $m$ is a hyperparameter, typically set to 1.0. The intuition behind the margin is that you want the model to rank the positive above the negative with confidence. A small score difference might indicate uncertainty, while a large margin suggests the model has learned a reliable distinction between the two documents. The margin also prevents the loss from rewarding overly conservative models that barely distinguish positive from negative examples.

Building on our understanding of the Bradley-Terry model from Bradley-Terry Model, we can also interpret pairwise preferences probabilistically. The probability that $d^+$ is preferred over $d^-$ is:

$$P(d^+ \succ d^- | q) = \sigma(s(q, d^+) - s(q, d^-))$$

where:

• $d^+ \succ d^-$: event showing document $d^+$ is preferred over document $d^-$
• $s(q, d^+)$: relevance score for the positive document
• $s(q, d^-)$: relevance score for the negative document
• $\sigma(\cdot)$: sigmoid function converting score differences to probabilities
• $P(d^+ \succ d^- | q)$: probability that $d^+$ is more relevant than $d^-$ given query $q$

The Bradley-Terry formulation says that preferences between items can be modeled through a latent "strength" parameter for each item. The probability that one item is preferred over another depends only on the difference in their strengths, transformed through a sigmoid. In our case, the relevance score $s(q, d)$ plays the role of strength, and we train the model to assign higher strength to relevant documents.

Maximizing the log-likelihood of observed preferences yields the logistic loss:

$$\mathcal{L} = -\sum \log \sigma(s(q, d^+) - s(q, d^-))$$

where:

• $s(q, d^+)$: relevance score for the positive document
• $s(q, d^-)$: relevance score for the negative document
• $\sigma(\cdot)$: sigmoid function
• $\mathcal{L}$: negative log-likelihood loss (logistic loss)

This formulation, used in RankNet and similar neural ranking models, naturally handles noisy preferences and provides well-calibrated probability estimates. Unlike the margin ranking loss, which treats preference violations with uniform penalty regardless of magnitude (beyond the margin), the logistic loss penalizes violations based on their probability, with larger mistakes receiving exponentially higher penalties. This probabilistic interpretation also allows you to combine evidence from multiple annotators or to model uncertainty in the preference labels themselves. A document pair that one annotator labels as $(d^+ \succ d^-)$ but another labels as $(d^- \succ d^+)$ can be treated as giving partial, uncertain preference evidence rather than a hard constraint.

While pairwise losses compare documents two at a time, listwise approaches consider the entire ranked list for a query. These methods often optimize target metrics more directly, such as Normalized Discounted Cumulative Gain (NDCG). The key insight behind listwise approaches is that ranking quality depends on individual document scores and on the relative ordering of all documents in the list. A pairwise approach might correctly order 99 pairs but still produce a suboptimal list if it makes a necessary error at the top position, while a listwise approach can directly optimize for the importance of top-ranked positions.

Think of listwise approaches as optimizing for the experience of someone who reads through the ranked list from top to bottom. The further down the list a reader goes, the less attention they typically pay to each result. Listwise losses incorporate this attention structure by weighting mistakes more heavily at higher positions. A relevant document ranked first is vastly more valuable than the same document ranked tenth, and listwise losses formalize this intuition mathematically.

One popular listwise loss is ListNet, which treats ranking as a probability distribution over permutations. For a list of documents $D = \{d_1, \dots, d_k\}$, define the probability of document $d_i$ being at the top using the softmax over scores:

$$P(d_i) = \frac{\exp(s(q, d_i))}{\sum_{j=1}^k \exp(s(q, d_j))}$$

where:

• $d_i$: the $i$-th document in the list of $k$ candidates
• $s(q, d_i)$: relevance score assigned to document $d_i$ by the model
• $k$: number of documents in the ranked list
• $\exp(\cdot)$: exponential function so positive values
• $P(d_i)$: probability that document $d_i$ should be ranked at the top position

The softmax formulation ensures that the probabilities sum to one across all documents in the list, creating a proper probability distribution over which document deserves the top position. Similarly, define target probabilities $P^*(d_i)$ based on relevance grades (e.g., exponential decay with position in the ideal ranking). The loss is the cross-entropy between these distributions:

$$\mathcal{L} = -\sum_{i=1}^k P^*(d_i) \log P(d_i)$$

where:

• $k$: number of documents in the list
• $P^*(d_i)$: target probability for document $d_i$ (derived from relevance grades)
• $P(d_i)$: predicted probability from the softmax over scores
• $\mathcal{L}$: cross-entropy loss between target and predicted ranking distributions

The target distribution $P^*(d_i)$ encodes the ideal ranking. If you have graded relevance labels (highly relevant, somewhat relevant, not relevant), you might set $P^*(d_i) \propto \exp(\text{grade}(d_i))$, so that highly relevant documents receive proportionally higher target probability. The cross-entropy loss then pushes the model's predicted distribution to match this target, training the model to assign high probability to the documents that should be at the top.

Listwise losses can better model the dependencies between documents and directly optimize for ranking quality, though they are computationally more intensive due to the need to process entire lists. They also allow you to incorporate position-dependent importance naturally, since the loss is sensitive to which documents appear at the top of the list. However, computing these losses over large lists can be expensive, and the softmax normalization can become numerically unstable with large score ranges. Variants like ListMLE simplify the computation by focusing on the probability of the observed permutation rather than the full distribution over all possible permutations.

A necessary aspect of training rerankers is negative sampling. If you train only on easy negatives (randomly sampled documents), the model learns little beyond what the bi-encoder already knows. The cross-encoder must learn to distinguish between documents that appear relevant to a fast retriever but are unsuitable for answering the query. Without hard negatives, the reranker fails to develop the discriminative capacity needed to improve upon the initial retrieval.

Think of easy negatives as training a chess player by having them practice against beginners. The beginner makes obvious blunders, and the student never needs to develop deep strategy. Hard negatives are like practice against near-equal opponents: the student must learn subtle principles and find the small advantages that separate winning from losing. A reranker trained only on easy negatives is like the chess player who can easily beat beginners but struggles against anyone who plays competently.

Instead, you should mine hard negatives: documents that the bi-encoder retrieves despite being irrelevant. These are documents that share semantic similarity with the query, perhaps containing overlapping vocabulary or related concepts, but fail to satisfy the specific information need. Hard negatives teach the model the subtle distinctions that separate plausible-looking documents from relevant ones.

The standard procedure is:

1. Retrieve top-$k$ documents for each training query using your bi-encoder
2. Filter out truly relevant documents (based on your labeled data)
3. Use the remaining retrieved documents as negatives

This forces the cross-encoder to learn the subtle distinctions that separate plausible-looking but irrelevant documents from relevant ones. As training progresses, you should periodically refresh the hard negatives using the current best retrieval model, a technique known as iterative hard negative mining. This iterative process creates a curriculum where the model progressively learns to distinguish finer and finer distinctions. Early in training, the model benefits from easier negatives to learn basic semantic distinctions. As the model improves, harder negatives force it to focus on increasingly subtle cues. However, care must be taken not to select negatives that are too hard: if a negative document is relevant (a false negative), training on it as a negative can degrade model performance. Quality control of negative mining is needed, often requiring human validation of edge cases or the use of multiple retrieval systems to identify likely true negatives.

The key insight is that the distribution of training negatives should mirror the distribution of errors you expect to encounter in production. Since the bi-encoder is your first-stage retriever in deployment, the documents it retrieves but should rank lower are exactly the errors you need the reranker to correct. Training on these errors creates a reranker that is specifically calibrated to improve upon the bi-encoder's weaknesses, which is precisely the objective of the two-stage architecture.