Contrastive Learning for Retrieval: InfoNCE & DPR Guide › Limitations and Impact
Language AI HandbookAll chapters Contrastive Learning for Retrieval: InfoNCE & DPR Guide › Limitations and ImpactOne section · Sections in this chapter10 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Contrastive learning for retrieval represented a watershed moment in information retrieval, proving that learned dense representations could match or exceed decades of tuned sparse methods like BM25. But the approach has important limitations that continue to drive research.
The most fundamental limitation is the dependence on negative quality and quantity. As we've seen, too-easy negatives waste training signal, while too-hard negatives (especially false negatives) can actively hurt the model. Getting the balance right requires careful engineering, and the optimal strategy varies across domains and datasets. Additionally, the strong batch size requirements mean that high-quality contrastive training demands significant compute resources, often 8 or more GPUs to achieve the batch sizes (128 or larger) where in-batch negatives become effective. This creates a meaningful barrier for researchers and practitioners without access to large GPU clusters.
A more subtle issue is that the dual-encoder architecture, while efficient at retrieval time, is fundamentally limited by the information bottleneck of the fixed-size embedding. A single 768-dimensional vector must capture everything about a passage that any possible query might ask about. This is why even well-trained dense retrievers sometimes miss relevant passages that a cross-encoder (which directly attends across query and passage tokens) would catch. This limitation motivates the reranking stage we'll discuss later in this part, where a more expensive cross-encoder re-scores the top candidates from the initial retrieval.
Despite these limitations, the contrastive learning framework has proven remarkably versatile. The same ideas power sentence embeddings (as we'll explore in the next chapter on embedding models), image-text matching (CLIP), code search, and multilingual retrieval. The DPR training recipe, specifically the combination of in-batch negatives, BM25 hard negatives, and InfoNCE loss, has become the starting point for nearly every modern dense retrieval system.