Loss Functions › Contrastive Losses

Contrastive losses are designed for representation learning tasks, where the goal is not to classify a fixed set of classes but to learn an embedding space where similar examples are close together and dissimilar examples are far apart.

The motivation comes from a fundamental limitation of cross-entropy: it requires a fixed set of predefined classes. But many real-world tasks do not fit that mold. In face verification, you want to check whether two photos show the same person, not classify the person into one of millions of identities. In semantic search, you want to retrieve documents similar to a query, not classify the query into a topic. In these settings, you need a loss that directly shapes the geometry of the embedding space, not one that categorizes examples into slots.

Contrastive learning has had enormous impact in NLP. Systems like SimCSE, DPR (Dense Passage Retrieval), and Sentence-BERT all use variants of contrastive objectives to learn sentence and passage representations that capture semantic similarity. The key insight is that you can construct training pairs (or triplets) from naturally occurring structure in data: paraphrases, question-answer pairs, adjacent sentences, or even a sentence and its dropout-augmented version.

The original contrastive loss (Hadsell et al., 2006) operates on pairs of examples $(x_i, x_j)$ with a binary label $y_{ij} \in \{0, 1\}$ indicating whether the pair is similar (1) or dissimilar (0):

$$\mathcal{L}_{\text{contrast}} = y_{ij} \cdot d_{ij}^2 + (1 - y_{ij}) \cdot \max(0, m - d_{ij})^2$$

where:

• $d_{ij} = \|\mathbf{e}_i - \mathbf{e}_j\|_2$: the Euclidean distance between the embeddings of examples $i$ and $j$
• $y_{ij}$: 1 if the pair is similar, 0 if dissimilar
• $m$: the margin hyperparameter, the minimum desired distance for dissimilar pairs
• $y_{ij} \cdot d_{ij}^2$: the loss for similar pairs, minimizing their distance
• $(1 - y_{ij}) \cdot \max(0, m - d_{ij})^2$: the loss for dissimilar pairs, penalizing only if they are closer than margin $m$

For similar pairs, the loss pushes embeddings together (minimizes $d_{ij}$). For dissimilar pairs, the loss pushes embeddings apart only when they are closer than the margin $m$. The margin prevents the model from uselessly separating pairs that are already far apart.

Triplet loss (Schroff et al., 2015, introduced for face recognition) uses triplets of examples: an anchor $a$, a positive example $p$ (same class or similar), and a negative example $n$ (different class or dissimilar):

$$\mathcal{L}_{\text{triplet}} = \max\left(0, d(a, p)^2 - d(a, n)^2 + \alpha\right)$$

where:

• $d(a, p)$: the distance between the anchor and positive embeddings
• $d(a, n)$: the distance between the anchor and negative embeddings
• $\alpha$: the margin hyperparameter, the minimum desired gap between positive and negative distances
• $\max(0, \cdot)$: the hinge function, applying zero loss when the constraint is already satisfied

Triplet loss ensures that the anchor is closer to its positive than to its negative by at least a margin $\alpha$. When the condition $d(a, n) > d(a, p) + \alpha$ is already satisfied, the loss is zero and no gradient flows. This means triplet mining is critical in practice: randomly sampled triplets are mostly "easy" (already satisfying the constraint), so training needs to focus on hard triplets where the negative is closer to the anchor than the positive.

Hard negative mining is the process of selecting, for each anchor, the negative example that is currently closest to the anchor in embedding space. These hard negatives provide the most informative gradient signal because they represent the cases the model is currently getting wrong. In NLP settings, hard negatives might be selected from examples that are semantically dissimilar to the query but lexically similar, which are the cases most likely to fool an embedding model.

Triplet loss is widely used for sentence embeddings, where semantically similar sentences should be close in embedding space. In the original Sentence-BERT paper, triplet loss with hard negatives trained on sentence pairs achieved far better results than training with pairwise or pointwise losses.

A modern extension of contrastive learning uses all other examples in the batch as negatives for each anchor, a technique that scales well and produces high-quality representations. The InfoNCE (Noise-Contrastive Estimation) loss treats the positive pair's similarity as the unnormalized score for a $K$-way classification problem over the $K$ examples in the batch:

$$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(\mathbf{a}, \mathbf{p}) / \tau)}{\sum_{j=1}^{K} \exp(\text{sim}(\mathbf{a}, \mathbf{e}_j) / \tau)}$$

where:

• $\mathbf{a}$: the anchor embedding
• $\mathbf{p}$: the positive embedding
• $\mathbf{e}_j$: the embedding of the $j$-th example in the batch (including the positive)
• $\text{sim}(\cdot, \cdot)$: cosine similarity between two embeddings
• $\tau$: a temperature hyperparameter controlling the sharpness of the distribution

InfoNCE scales naturally to large batches: with a batch of 256 examples, each anchor sees 255 in-batch negatives. This provides rich training signal without any explicit negative sampling strategy. SimCSE uses InfoNCE with dropout-augmented positives (the same sentence encoded twice with different dropout masks) and achieves state-of-the-art sentence embedding quality.

The temperature $\tau$ controls the strength of the InfoNCE discrimination signal. Low temperature (around 0.05 to 0.1) makes the loss more sensitive to the relative order of similarities, pushing the model to create sharp distinctions between the positive and all negatives. High temperature makes the task easier but also reduces the discrimination signal. In practice, temperature is treated as a hyperparameter to tune, or learned as a trainable parameter initialized near 0.07.

A class of losses closely related to contrastive learning is designed specifically for information retrieval and ranking problems. In ranking, the goal is not to assign correct class labels but to order items so that the most relevant items appear first.

Pairwise ranking loss (also called margin ranking loss) requires the score of the positive example to exceed the score of the negative example by at least a margin:

$$\mathcal{L}_{\text{pair}} = \max(0, \text{margin} - s_{\text{pos}} + s_{\text{neg}})$$

where $s_{\text{pos}}$ and $s_{\text{neg}}$ are scalar relevance scores for the positive and negative items respectively. This is structurally identical to the linear hinge loss, applied to ranking.

Listwise losses operate on the entire ranked list at once, optimizing directly for ranking metrics like NDCG or MAP. ListNet and LambdaRank are examples. These are more complex to implement but can directly optimize the ranking metric of interest rather than a surrogate.

In NLP, ranking losses appear in dense passage retrieval (DPR), where the goal is to rank relevant passages above non-relevant ones for a given query. The DPR model uses a variant of in-batch negatives similar to InfoNCE, treating each passage in the batch as a potential negative for every query in the batch.