Contrastive Learning for Retrieval: InfoNCE & DPR Guide › Summary

Contrastive learning trains retrieval models by teaching them to distinguish relevant passages from irrelevant ones. The key concepts covered in this chapter are:

• InfoNCE loss frames retrieval as multi-class classification over similarity scores. The model must assign the highest probability to the true positive passage among all candidates. A temperature parameter $\tau$ controls the sharpness of the softmax distribution.

• In-batch negatives reuse the positive passages of other queries in a mini-batch as negatives, producing $B - 1$ negatives for free from a batch of $B$ examples. This makes batch size a critical hyperparameter: larger batches provide more negatives and consistently yield better models.

• Hard negative mining provides challenging examples that share surface features with the positive but are irrelevant. BM25 retrieval is the most common source, since it finds lexically similar passages that lack true semantic relevance. Mixing hard negatives with easier in-batch negatives gives the best training dynamics.

• DPR's training procedure combines these ideas into a complete system: two BERT-base encoders (one for queries, one for passages), trained with InfoNCE loss over in-batch negatives plus one BM25 hard negative per query, at batch size 128 across 8 GPUs.

The next chapter on document chunking will address a practical prerequisite for applying these retrieval models: how to split long documents into passages that are the right size for encoding and retrieval.