Contrastive Learning for Retrieval: InfoNCE & DPR Guide › DPR Training Procedure

Dense Passage Retrieval (DPR), introduced by Karpukhin et al. in 2020, was one of the first systems to demonstrate that a contrastive-trained dense retriever could outperform BM25 on open-domain question answering. Its training procedure combines the techniques we've discussed into a coherent pipeline and is the template for most subsequent dense retrieval models. By examining DPR in detail, we can see how the abstract ideas of contrastive loss, in-batch negatives, and hard negative mining come together into a working system.

DPR uses a dual-encoder architecture with two independent BERT-base models (as we covered in Part XXIV): one for queries and one for passages. Each encoder takes its input text, processes it through 12 transformer layers, and produces a $d = 768$-dimensional embedding by taking the [CLS] token representation from the final layer. This [CLS] embedding is a single fixed-size summary of the entire input sequence, compressing all the information in a query or passage into a vector that can be compared via simple arithmetic.

The choice to use separate encoders for queries and passages is deliberate and reflects the asymmetric nature of the retrieval task. Queries are typically short (5-15 tokens) while passages are longer (up to 256 tokens), and their linguistic properties differ substantially. A query is often a question or a fragmentary phrase, while a passage is a coherent piece of expository text. Separate encoders allow each to specialize: the query encoder can learn to interpret the intent behind terse questions, while the passage encoder can learn to extract the key information from longer prose. However, both encoders share the same architecture (BERT-base), just initialized from the same pretrained checkpoint and then fine-tuned independently through the contrastive training process.

The similarity function is a simple dot product:

$$\text{sim}(q, p) = E_Q(q)^\top E_P(p)$$

where:

• $\text{sim}(q, p)$: the similarity score used by DPR
• $E_Q(q)$: the embedding output of the query encoder
• $E_P(p)$: the embedding output of the passage encoder
• $\top$: the transpose operation, computing the dot product between the two vectors

No learned projection layers, no cosine similarity normalization. The dot product is chosen for its computational simplicity at retrieval time: computing a dot product between two 768-dimensional vectors requires only 768 multiplications and 767 additions, making it fast enough to score millions of candidates per second with appropriate indexing. This efficiency is essential because at inference time, the retriever must search over the entire corpus, which may contain millions of passages, to find the top-$k$ most relevant results. We'll explore the indexing techniques that make this feasible in the chapters on vector similarity search and indexing.

DPR was trained on five question-answering datasets: Natural Questions, TriviaQA, WebQuestions, CuratedTREC, and SQuAD. For each question, the positive passage is the paragraph containing the answer. The training set contains roughly 60,000 question-passage pairs. This is a relatively modest dataset by modern standards, but the contrastive training framework with in-batch negatives extracts a surprisingly large amount of training signal from each example, because every passage in the batch is a negative for every other query.

The negative passages for each question come from three sources, combined into a single training example:

• In-batch negatives: The positive passages from all other questions in the mini-batch ($B - 1$ negatives). These provide a large, diverse set of mostly easy negatives that give the softmax distribution its broad shape.
• One BM25 hard negative: The highest-scoring BM25 passage that doesn't contain the answer string. This provides a lexically similar but non-answering passage, forcing the model to learn that keyword overlap does not equal relevance.
• One gold negative (when available): A passage annotated by humans as relevant-looking but incorrect. These are the highest-quality negatives in the training set. This provides a clear signal about exactly the kind of mistake the model should avoid.

This mixture is carefully chosen, and the DPR paper includes ablation studies showing that each component contributes to the final performance. The in-batch negatives provide a large number of diverse (mostly easy) negatives for a rich softmax distribution. This keeps the gradients are stable and well-behaved. The BM25 hard negative adds one challenging example that forces the model beyond lexical matching, directly addressing the failure mode of relying on surface features. The combination proved more effective than using any single negative strategy alone, confirming the principle that a diverse mix of negative difficulties produces the best training dynamics.

The DPR loss is a direct application of InfoNCE. For a batch of $B$ questions, with each question having one positive passage, one BM25 hard negative, and $B - 1$ in-batch negatives, the loss for question $q_i$ is:

$$\mathcal{L}(q_i) = -\log \frac{\exp\bigl(\text{sim}(q_i, p_i^+)\bigr)}{\exp\bigl(\text{sim}(q_i, p_i^+)\bigr) + \exp\bigl(\text{sim}(q_i, p_i^{\text{BM25}})\bigr) + \sum_{j \neq i} \exp\bigl(\text{sim}(q_i, p_j^+)\bigr)}$$

where:

• $\mathcal{L}(q_i)$: the loss for the $i$-th query in the batch
• $\text{sim}(q_i, p_i^+)$: similarity with the true positive passage
• $\text{sim}(q_i, p_i^{\text{BM25}})$: similarity with the BM25 hard negative
• $\sum_{j \neq i} \exp\bigl(\text{sim}(q_i, p_j^+)\bigr)$: sum of exponentials of similarities with in-batch negatives (positives from other queries)
• $\exp(\cdot)$: the exponential function ensuring positive values
• $-\log$: the negative log-likelihood maximizing the probability of the positive pair

To understand this formula concretely, consider what the denominator contains. There are three groups of terms. The first term, $\exp(\text{sim}(q_i, p_i^+))$, is the positive passage's contribution. The second term, $\exp(\text{sim}(q_i, p_i^{\text{BM25}}))$, is the BM25 hard negative's contribution, representing the single most challenging distractor. The remaining terms in the summation, $\sum_{j \neq i} \exp(\text{sim}(q_i, p_j^+))$, are the contributions from all other passages in the batch. The model must push the numerator (the positive) to dominate this entire denominator, which means simultaneously pushing the positive score up and all negative scores down.

Note that DPR uses $\tau = 1$ (no temperature scaling). The temperature is effectively absorbed into the magnitude of the learned embeddings: since there is no normalization step, the model is free to learn embeddings of whatever magnitude is needed to produce well-calibrated softmax distributions. The total batch loss averages over all $B$ questions, so each training step updates the encoders based on the collective signal from every query-passage interaction in the batch.

The specific training configuration that DPR found effective includes:

• Batch size: 128 (using 8 GPUs with 16 examples per GPU, sharing negatives across GPUs to get the full 127 in-batch negatives).
• Learning rate: $2 \times 10^{-5}$ with linear warmup and linear decay.
• Epochs: 40 epochs over the training data.
• Sequence lengths: 256 tokens for passages, 32 tokens for queries.
• Optimizer: Adam, consistent with standard BERT fine-tuning practices we discussed in Part XXXIV.

The large batch size is critical and deserves special emphasis. The DPR paper's ablation studies showed that increasing batch size from 16 to 128 improved top-20 retrieval accuracy by over 3 percentage points, directly demonstrating the importance of having many in-batch negatives. This improvement separates a system that is competitive with BM25 from one that clearly surpasses it. The choice of 128 was not arbitrary but reflected the largest batch size achievable on the available hardware (8 V100 GPUs), and subsequent work has shown that even larger batch sizes can yield further improvements when the compute budget allows.