Reranking: Cross-Encoders for Precise Information Retrieval › Cross-Encoder Architecture

Cross-encoders eliminate the independence assumption by processing query and document jointly. Rather than compressing each into a vector separately, they concatenate the texts and feed them through a transformer, letting every query token to attend directly to every document token. This joint processing enables the model to perform fine-grained comparisons that are impossible when representations are computed in isolation. When the query token "temperature" can directly attend to the document token "32°C," the model can verify the presence of specific information rather than merely estimating semantic overlap.

The architecture is conceptually simple but computationally powerful. You take a pre-trained transformer encoder, the same kind of BERT model we examined in BERT Architecture, and fine-tune it on relevance-labeled query-document pairs. The transformer's self-attention mechanism, applied across the concatenated input, creates a rich interaction matrix between all query and document tokens. Every query token can ask "Is this information present in the document?" and receive an answer informed by the full document context. Every document token can ask "Does this piece of information satisfy the query?" and receive an answer informed by the full query intent.

The standard cross-encoder format concatenates the query and document with special separator tokens:

$$\text{input} = [\text{CLS}] \oplus \text{query} \oplus [\text{SEP}] \oplus \text{document} \oplus [\text{SEP}]$$

where:

• $[\text{CLS}]$: classification token marking the start of the sequence (following BERT conventions from BERT Architecture)
• $\text{query}$: token sequence of the query
• $\text{document}$: token sequence of the document
• $[\text{SEP}]$: separator token marking boundaries between segments
• $\oplus$: token sequence concatenation operator

The entire sequence is processed through a transformer encoder, typically BERT or a similar masked language model. The use of special separator tokens is important because it allows the transformer to distinguish between query and document content. Without these explicit boundaries, the model would struggle to determine which part of the concatenated sequence represents your information need versus the potential source of that information. The [CLS] token, originally introduced for next-sentence prediction tasks in BERT, is an aggregate representation of the interaction between the two texts.

The segment embeddings that BERT uses for next-sentence prediction tasks serve a secondary role here. Segment A typically corresponds to the query, and segment B corresponds to the document. These segment embeddings provide the model with a structural prior about which text is the "question" and which is the "answer," even before the attention mechanism processes the content. In practice, models fine-tuned specifically for passage reranking often discard the segment embeddings or treat the entire concatenation as a single segment, finding that the [SEP] token alone is sufficient to signal the boundary.

After processing through the transformer layers, the final hidden state of the [CLS] token, $\mathbf{h}_{\text{CLS}} \in \mathbb{R}^d$, is the pooled representation of the query-document pair. This single vector condenses all the pairwise interactions between query and document tokens into a fixed-size summary. A scoring head, typically a simple feed-forward network or even a linear projection, maps this to a relevance score:

$$s(q, d) = \mathbf{w}^T \mathbf{h}_{\text{CLS}} + b$$

where:

• $\mathbf{h}_{\text{CLS}} \in \mathbb{R}^d$: the final hidden state of the [CLS] token, serving as the pooled representation
• $\mathbf{w} \in \mathbb{R}^d$: weight vector of the scoring head
• $b \in \mathbb{R}$: bias term of the scoring head
• $s(q, d)$: the predicted relevance score (logit) for query-document pair $(q, d)$

The [CLS] token is an interesting choice for pooling. During pre-training, BERT trained this token to predict whether two sentences were consecutive, which forced it to encode the relationship between the two input segments. Fine-tuning on relevance prediction adapts this representational capacity to encode query-document compatibility rather than sentence adjacency. Alternative pooling strategies, such as mean-pooling over all tokens or max-pooling over position-wise representations, are possible but empirically the [CLS] approach works well, especially when the model is initialized from a BERT checkpoint that already has strong [CLS]-based representations.

For classification-based training, the score is often passed through a sigmoid to produce a probability of relevance:

$$P(\text{relevant} | q, d) = \sigma(s(q, d)) = \frac{1}{1 + \exp(-s(q, d))}$$

where:

• $s(q, d)$: the relevance score (logit) computed by the scoring head
• $\sigma(\cdot)$: the sigmoid function, mapping logits to probabilities in $(0, 1)$
• $P(\text{relevant} | q, d)$: probability that document $d$ is relevant to query $q$

The sigmoid transformation is necessary when you need calibrated probability estimates rather than ordinal rankings. For a well-calibrated reranker, $P(\text{relevant} | q, d) = 0.8$ means that documents assigned this score are relevant 80% of the time. Calibration matters for downstream decisions like whether to include a document in the context window for generation, or whether to return "no relevant document found" rather than returning the top result by default.

Alternatively, some rerankers output logits for multiple relevance grades (e.g., not relevant, somewhat relevant, highly relevant) using a softmax over multiple output classes. This graded approach allows the model to express degrees of relevance rather than binary judgments, which can be particularly useful when training on data with multiple relevance levels, such as human judgments on a three-point or five-point scale. The choice between regression, binary classification, or ordinal classification depends on the nature of your training data and the specific requirements of your ranking task.

The richness of interaction comes at steep computational cost. While bi-encoders require $O(L_q + L_d)$ transformer passes per query-document pair, reduced to $O(L_d)$ if documents are pre-encoded, cross-encoders require $O((L_q + L_d)^2)$ attention computations per pair.

Specifically, for a sequence of length $L = L_q + L_d + \text{special tokens}$, the self-attention mechanism computes:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

where:

• $Q \in \mathbb{R}^{L \times d_k}$: query matrix representing projected input tokens
• $K \in \mathbb{R}^{L \times d_k}$: key matrix representing projected input tokens
• $V \in \mathbb{R}^{L \times d_k}$: value matrix representing projected input tokens
• $L$: sequence length (total tokens in query + document + special tokens)
• $d_k$: dimension of the key/query projections
• $\text{softmax}(\cdot)$: normalization function converting attention scores to probabilities

The matrix multiplication $QK^T$ alone costs $O(L^2 d_k)$. If your query is 10 tokens and your document is 512 tokens, a bi-encoder processes them separately (cost proportional to $10^2 + 512^2 \approx 262,000$), while a cross-encoder processes the concatenation (cost proportional to $522^2 \approx 272,000$). However, since bi-encoders pre-encode documents, the per-query cost is just $O(10^2)$ for the query encoding plus $O(1)$ for the dot product, making them thousands of times faster at query time.

This computational asymmetry fundamentally shapes system architecture. While the bi-encoder pays a high upfront cost to encode the corpus (which can be amortized over billions of queries), the cross-encoder must perform expensive computation for every query-document pair at query time. This means that if you attempt to rerank thousands of documents per query, latency becomes prohibitive, often reaching several seconds or even minutes. The quadratic scaling with sequence length also means that long documents dramatically increase computational cost, creating pressure to truncate texts aggressively, which risks losing relevant information located in later portions of the document.

Think of the computational cost as a restaurant analogy. A bi-encoder is like a restaurant that prepares all the dishes in advance and stores them in a display case. When a customer arrives, serving them is nearly instantaneous. A cross-encoder is like a restaurant that cooks every dish to order, personalized for each specific customer. The food is better, but every customer has to wait. The two-stage architecture is like a restaurant that prepares a large batch of semi-finished dishes in advance, then does a final personalized preparation for each customer from a shortlist. You get quality close to fully custom-cooked, at a speed close to the display case.