Reranking: Cross-Encoders for Precise Information Retrieval › Key Parameters

The key parameters for the retrieve-then-rerank implementation deserve careful consideration, as they fundamentally shape the behavior and performance of your system. Choosing these parameters correctly requires understanding both the mathematical properties of the models and the empirical constraints of your deployment environment.

The most important parameters are:

• k (reranking budget): Number of candidates to retrieve for reranking. Higher values improve recall for the reranker but increase latency linearly. Selecting the appropriate value for $k$ requires balancing the probability that the relevant document appears in the top $k$ (recall) against the latency constraints of your application. In practice, $k$ is often determined empirically by measuring the recall@k of the initial retriever on a validation set and choosing the smallest $k$ that achieves acceptable recall. A common heuristic is to set $k$ ten to twenty times larger than the number of final results you want to return.

• convert_to_tensor: Whether to return PyTorch tensors for the embeddings, letting GPU acceleration for similarity computations. Using GPU acceleration for both bi-encoder and cross-encoder operations is needed for achieving acceptable latency in production systems. The tensor format also facilitates batch processing, which can significantly improve throughput when processing multiple queries simultaneously.

• model: The specific cross-encoder architecture (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2). Larger models provide better accuracy at the cost of increased inference time. Model selection involves trading off between the improved discrimination of larger architectures (such as large BERT or RoBERTa variants) and the latency requirements of your use case. Models trained on MS MARCO, a large-scale passage ranking dataset, generally provide strong zero-shot performance for general-domain retrieval tasks.

• max_length: Maximum sequence length for the cross-encoder (typically 512 tokens). Documents longer than this are truncated, potentially losing relevant information at the end. When working with long documents, consider using the sliding window approach discussed earlier, or select models specifically designed for long sequences (such as those using Longformer or BigBird architectures) that can handle thousands of tokens.