Contrastive Learning for Retrieval: InfoNCE & DPR Guide › Contrastive Loss Functions

Several loss functions formalize the contrastive objective, each encoding slightly different assumptions about what it means for a model to "succeed" at distinguishing positives from negatives. We'll cover the two most important ones for retrieval: the triplet loss and The InfoNCE loss (also called the Normalized Temperature-scaled Cross-Entropy loss). Understanding both is valuable because the triplet loss illustrates the core geometric intuition clearly, while the InfoNCE loss shows how to scale that intuition into a practical training signal using many negatives at once.

The triplet loss is the most geometrically intuitive contrastive objective. It operates on a single triplet: an anchor (the query), a positive, and a negative. The idea is straightforward: the model should place the anchor closer to the positive than to the negative by at least a fixed margin $m$, not merely by an infinitesimal amount. The loss penalizes the model whenever this margin condition is violated:

$$\mathcal{L}_{\text{triplet}} = \max\bigl(0,\; s(q, p^-) - s(q, p^+) + m\bigr)$$

where:

• $\mathcal{L}_{\text{triplet}}$: the triplet loss value
• $s(q, p^+)$: the similarity score between the query and the positive passage
• $s(q, p^-)$: the similarity score between the query and the negative passage
• $m$: the margin, a hyperparameter ensuring the positive is significantly closer than the negative
• $\max(0, \cdot)$: the hinge function, which is zero if the condition $s(q, p^+) \ge s(q, p^-) + m$ is met

To build intuition for how this formula works, consider what happens inside the $\max$ function. The expression $s(q, p^-) - s(q, p^+) + m$ measures how much the model is failing to maintain the desired margin. If the positive passage already has a similarity score that exceeds the negative's score by at least $m$, then $s(q, p^-) - s(q, p^+)$ is at most $-m$, and adding $m$ brings the value to zero or below. The $\max(0, \cdot)$ clamps this to zero, meaning the loss contributes nothing and the model receives no gradient from this triplet. The model has already "solved" this example. If, however, the positive is not sufficiently ahead of the negative, the expression inside the $\max$ is positive, and the loss is exactly the amount by which the margin condition is violated. The gradient then pushes the positive score up and the negative score down until the margin is restored.

The margin $m$ is a critical design choice. A small margin (e.g., 0.1) asks only that the positive be slightly more similar than the negative, which may result in embeddings that work for easy cases but fail on harder ones. A large margin (e.g., 1.0) demands a wide gap between positive and negative scores, producing more discriminative embeddings but potentially making training harder if the model cannot easily achieve such separation early in training. In practice, $m$ is typically tuned as a hyperparameter alongside other training settings.

The triplet loss is intuitive but has a significant limitation: it only considers one negative at a time. This means the model gets a very noisy gradient signal, since each update only pushes the model away from a single irrelevant passage. If you happen to sample an easy negative, the loss is zero and no learning occurs on that step. If you sample a hard negative, the gradient may be large but unrepresentative of the broader set of negatives the model will encounter at retrieval time. In practice, training with triplet loss converges slowly and is sensitive to how you sample triplets, often requiring careful mining strategies just to achieve reasonable performance.

The InfoNCE loss (also called NTXent or multi-class cross-entropy loss over similarities) addresses the triplet loss's limitation by considering all negatives simultaneously. Rather than asking the model a binary question ("is the positive closer than this one negative?"), InfoNCE asks a much richer question: "among all these candidates, can you identify which one is the true positive?" This reformulation turns each training example into a much more informative learning signal.

Given a query $q$, its positive $p^+$, and $N$ negatives $\{p_1^-, \ldots, p_N^-\}$, the loss treats retrieval as an $(N+1)$-way classification problem: which of the $N+1$ candidates is the correct positive?

$$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp\bigl(s(q, p^+) / \tau\bigr)}{\exp\bigl(s(q, p^+) / \tau\bigr) + \sum_{i=1}^{N} \exp\bigl(s(q, p_i^-) / \tau\bigr)}$$

where:

• $\mathcal{L}_{\text{InfoNCE}}$: the total loss value to be minimized
• $s(q, p^+)$: the similarity score between the query and the positive passage
• $s(q, p_i^-)$: the similarity score between the query and the $i$-th negative passage
• $N$: the total number of negative passages
• $\tau$: the temperature parameter scaling the logits
• $\sum_{i=1}^{N}$: the summation over all negative samples in the denominator
• $\exp(\cdot)$: the exponential function ensuring positive values
• $-\log$: the negative log-likelihood maximizing the probability of the positive pair

This should look familiar, and understanding why requires us to unpack the formula layer by layer. The fraction inside the logarithm is a softmax probability. The numerator computes the exponentiated (and temperature-scaled) similarity between the query and the true positive passage. The denominator sums that same quantity together with the exponentiated similarities for all $N$ negatives. The resulting ratio is therefore the probability that the model assigns to the positive passage being the correct match, given the full set of $N+1$ candidates. Taking the negative logarithm converts this probability into a loss: if the model assigns probability 1.0 to the positive, the loss is zero; as the probability assigned to the positive decreases, the loss grows, penalizing the model more heavily.

The structure is identical to the cross-entropy loss we discussed in Part X, but applied to similarity scores rather than logit outputs from a classification head. In a standard classification setting, the model produces a vector of logits for each class, and cross-entropy penalizes the model for not placing the highest probability on the correct class. Here, the "classes" are the candidate passages, and the "logits" are the similarity scores between the query and each candidate. The key insight is recognizing that retrieval can be framed as classification, where the "correct class" is the relevant passage.

The temperature parameter $\tau$ controls the sharpness of the softmax distribution, and understanding its role is essential for building good intuition about contrastive training dynamics. A small $\tau$ (e.g., 0.05) makes the distribution peaky, amplifying differences between scores and forcing the model to be very confident about which candidate is the positive. A larger $\tau$ (e.g., 1.0) produces a flatter distribution, making the task easier but potentially learning less discriminative representations. This is the same temperature concept we saw in Part XXVIII when discussing decoding temperature for text generation, but applied here during training rather than inference.

Let's build intuition for why $\tau$ matters by tracing through a concrete numerical example. With $\tau = 1.0$, similarity scores pass through the softmax relatively unchanged. A score of 0.7 for the positive and 0.6 for a negative yields a modest difference after exponentiation: $\exp(0.7) \approx 2.01$ versus $\exp(0.6) \approx 1.82$. But with $\tau = 0.05$, those same scores become $\exp(0.7/0.05) = \exp(14) \approx 1.2 \times 10^6$ versus $\exp(0.6/0.05) = \exp(12) \approx 1.6 \times 10^5$. A score difference of 0.1 between the positive and the best negative gets amplified by a factor of 20 before the softmax, which turns what was a gentle preference into an overwhelming one. This forces the model to create larger gaps between positive and negative scores. This produces more discriminative embeddings. The tradeoff is that very small temperatures can make training unstable, because the gradients become extremely sensitive to small changes in similarity scores. In practice, the temperature is often treated as a hyperparameter to be tuned, and some models even learn it during training.

The InfoNCE loss has become the standard for contrastive retrieval training for several interconnected reasons, each of which illuminates a different aspect of what makes a good training objective:

• Richer gradient signal: Each training example provides information about how the query relates to all candidates simultaneously, not just one negative. With the triplet loss, a single training step only says "be closer to the positive than to this one negative." With InfoNCE, a single training step says "be closer to the positive than to all of these negatives," and the softmax probability distribution encodes the relative difficulty of each negative. Negatives that the model already handles well contribute little to the gradient, while negatives that the model confuses with the positive contribute heavily. This natural weighting of the gradient is far more efficient than the all-or-nothing signal of the triplet loss.
• Natural probabilistic interpretation: The loss directly models $P(\text{positive} \mid q, \text{candidates})$, making it a proper scoring function. This means the learned similarity scores have a meaningful interpretation: they are proportional to the log-probability that a passage is relevant to the query, given a set of alternatives. This probabilistic grounding connects the training objective to the downstream task of ranking, where you want the highest-scoring passage to be the most relevant one.
• Scales with negatives: As we increase $N$, the task becomes harder and the learned representations become more discriminative. Research consistently shows that more negatives lead to better retrieval quality, up to a point. This scaling property is unique to the multi-class formulation; the triplet loss, by contrast, does not naturally benefit from having more negatives because it processes them one at a time.

The connection between number of negatives and representation quality is not coincidental; it reflects a deep theoretical property of the InfoNCE objective. Information-theoretically, InfoNCE provides a tighter lower bound on the mutual information between queries and relevant passages as $N$ grows. More negatives mean the model must learn finer distinctions, which directly translates to better retrieval. Intuitively, if you have to pick the correct passage from among 5 candidates, you can get away with coarse features. But if you must pick the correct passage from among 500 candidates, many of which are topically similar, you need representations that capture subtle semantic differences. The InfoNCE objective with large $N$ naturally induces this pressure.