Reranking: Cross-Encoders for Precise Information Retrieval › Limitations and Impact

While cross-encoder reranking dramatically improves retrieval accuracy, it introduces significant operational constraints that shape system architecture. The quadratic complexity with respect to sequence length means that processing long documents becomes prohibitively expensive. You cannot easily rerank entire books or lengthy research papers without aggressive truncation or chunking, which risks losing relevant information buried in the middle or end of texts. This limitation has driven research into efficient transformers and hierarchical approaches that can process long documents without quadratic scaling. The practical consequence is that most reranking systems operate on chunks or passages rather than full documents, which shifts complexity from the reranking step to the document preprocessing and chunking strategy.

The lack of precomputation also requires running a forward pass through a large transformer for every query-document pair at query time, creating a throughput bottleneck that limits the number of candidates you can rerank within latency budgets. Unlike bi-encoders, which amortize document encoding over all queries, cross-encoders must reprocess documents for every new query. This architectural constraint means that cross-encoders scale with query volume in ways that can strain GPU resources during traffic spikes. A system that works well for ten queries per second may fail catastrophically at one hundred queries per second if the GPU cannot keep up with the reranking demands. Horizontal scaling requires careful load balancing and may require maintaining multiple replicas of the cross-encoder model, increasing infrastructure costs significantly.

Cross-encoders also inherit the context length limitations of the underlying transformer. Standard BERT-based models process at most 512 tokens, which is sufficient for short passages but inadequate for longer documents without chunking. Even extended-context models face practical limitations: processing a 4,096-token document requires roughly 64 times more compute than processing a 512-token document due to quadratic attention scaling. This creates asymmetric behavior where the reranker performs well on short, focused passages but degrades on longer, more complex documents that may require evidence from across the full text to judge relevance.

These limitations have spurred the development of intermediate architectures like ColBERT and SPLADE that attempt to preserve token-level interaction capabilities while maintaining indexable representations. They also motivate the cascade approaches mentioned earlier, where you progressively filter candidates through increasingly expensive models. Reranking established the two-stage retrieve-then-rerank paradigm as the standard for neural information retrieval, allowing systems to balance recall and precision within a latency budget. Without reranking, dense retrieval systems would struggle to achieve the accuracy required for production search engines and retrieval-augmented generation applications, where returning an irrelevant document in the top position significantly degrades user experience.

Looking ahead, the reranking paradigm is increasingly intersecting with large language models as judges. Rather than using a fine-tuned cross-encoder, some recent approaches use a general-purpose LLM to assess the relevance of each retrieved document by prompting it with the query and document and asking it to rate relevance on a numerical scale. This uses the LLM's language understanding without requiring task-specific fine-tuning data, though it comes with even higher latency costs than traditional cross-encoders. As we move toward more sophisticated RAG pipelines in upcoming chapters on RAG Prompt Engineering and Evaluation, the reranker serves as a quality filter so that only the most relevant context reaches the language generator. Selecting the right passage can mean the difference between a correct and incorrect answer, making the reranker one of the most consequential components in the RAG system.