Contrastive Learning for Retrieval: InfoNCE & DPR Guide › Contrastive Learning for Retrieval

In the previous chapter, we introduced dense retrieval and the idea of encoding queries and passages into a shared embedding space where relevance is measured by vector similarity. But we left a critical question unanswered: how do you train encoders to place relevant query-passage pairs close together and irrelevant ones far apart?

The answer is contrastive learning, a training paradigm built around a simple idea: teach a model what things should look like together by also showing it what things should not look like together. Rather than predicting labels or generating text, a contrastive model learns to distinguish positive pairs (a query matched with its relevant passage) from negative pairs (a query matched with irrelevant passages). The loss function rewards the model for increasing the similarity of positive pairs while decreasing the similarity of negatives.

This idea has roots we've seen before. Recall from Part IV that negative sampling in Word2Vec works on a similar principle: the skip-gram model learns word embeddings by distinguishing true context words from randomly sampled negative words. Contrastive learning for retrieval extends this concept to full sequences, using BERT-style encoders instead of lookup tables and operating over passages rather than individual words.

In this chapter, we'll build up the contrastive learning framework piece by piece. We start with the core loss function, then examine two techniques that make training practical and effective: in-batch negatives (which give you hundreds of negatives "for free") and hard negative mining (which provides the challenging examples a model needs to learn fine-grained distinctions). We'll then walk through the complete training procedure of Dense Passage Retrieval (DPR), the model that brought these ideas together into a system that outperformed BM25 on open-domain question answering.