Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

CLIP › Worked Example: Computing the Contrastive Loss

Michael BrenndoerferFebruary 8, 20265 min read

Language AI HandbookAll chapters

Part 50 of 67Multimodal Models

Chapter 382CLIP › Worked Example: Computing the Contrastive Loss

Reading section 6 of 11 · Worked Example: Computing the Contrastive Loss · 5 minSection 6 of 11 · 5 min

Section 6 of 11 · 5 min read

CLIP › Worked Example: Computing the Contrastive LossOne section · Sections in this chapter6 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's trace through a minimal example to solidify the mechanics. Suppose we have a batch of $N = 3$ image-text pairs:

Image 1: Dog, Text: "a dog playing fetch"
Image 2: Cat, Text: "a cat sleeping on a couch"
Image 3: Bird, Text: "a bird in flight"

After encoding and L2 normalization, we obtain cosine similarities. For this simplified illustration, we treat these values as the scaled similarity scores $S_{ij}$ (assuming a temperature of $\tau = 1$ for clarity):

Similarity matrix for the worked example with three image-text pairs. Diagonal values (correct pairs) are high, while off-diagonal values (incorrect pairs) are low.
	Text 1	Text 2	Text 3
Image 1	0.9	0.1	0.2
Image 2	0.2	0.85	0.15
Image 3	0.1	0.3	0.88

The diagonal represents correct pairs. To compute the image-to-text loss for Image 1, we calculate the softmax probability of the correct match:

\begin{aligned} P(\text{Text 1} | \text{Image 1}) &= \frac{\exp(0.9)}{\exp(0.9) + \exp(0.1) + \exp(0.2)} \\ &\approx \frac{2.46}{2.46 + 1.11 + 1.22} \\ &\approx 0.514 \end{aligned}

where:

$P(\text{Text 1} | \text{Image 1})$ : the softmax probability that Text 1 is the correct match for Image 1
$0.9$ : the cosine similarity score $S_{11}$ between Image 1 and Text 1 (the correct pair)
$0.1$ : the cosine similarity score $S_{12}$ between Image 1 and Text 2 (an incorrect pair)
$0.2$ : the cosine similarity score $S_{13}$ between Image 1 and Text 3 (an incorrect pair)
$2.46, 1.11, 1.22$ : the exponential values $e^{0.9}$ , $e^{0.1}$ , and $e^{0.2}$ respectively
$0.514$ : the final probability value (approximately 51.4%)

The loss contribution is $-\log(0.514) \approx 0.665$ .

For Image 2:

\begin{aligned} P(\text{Text 2} | \text{Image 2}) &= \frac{\exp(0.85)}{\exp(0.2) + \exp(0.85) + \exp(0.15)} \\ &\approx \frac{2.34}{1.22 + 2.34 + 1.16} \\ &\approx 0.495 \end{aligned}

where:

$P(\text{Text 2} | \text{Image 2})$ : the softmax probability that Text 2 is the correct match for Image 2
$0.85$ : the cosine similarity score $S_{22}$ between Image 2 and Text 2 (the correct pair)
$0.2$ : the cosine similarity score $S_{21}$ between Image 2 and Text 1 (an incorrect pair)
$0.15$ : the cosine similarity score $S_{23}$ between Image 2 and Text 3 (an incorrect pair)
$2.34, 1.22, 1.16$ : the exponential values $e^{0.85}$ , $e^{0.2}$ , and $e^{0.15}$ respectively
$0.495$ : the final probability value (approximately 49.5%)

Loss contribution: $-\log(0.495) \approx 0.703$ .

For Image 3:

\begin{aligned} P(\text{Text 3} | \text{Image 3}) &= \frac{\exp(0.88)}{\exp(0.1) + \exp(0.3) + \exp(0.88)} \\ &\approx \frac{2.41}{1.11 + 1.35 + 2.41} \\ &\approx 0.495 \end{aligned}

Loss contribution: $-\log(0.495) \approx 0.703$ .

Averaging the three image-to-text losses:

\mathcal{L}_{\text{i2t}} = \frac{1}{3}(0.665 + 0.703 + 0.703) \approx 0.690

The text-to-image losses are computed symmetrically by treating each text as a query and computing the softmax across image similarities (summing down columns rather than across rows). The final CLIP loss averages both directions: $\mathcal{L}_{\text{CLIP}} = \frac{1}{2}(\mathcal{L}_{\text{i2t}} + \mathcal{L}_{\text{t2i}})$ .

Notice that with only three items in the batch, the model achieves about 51% probability on the correct pair. In a large batch with 32,768 items, achieving 51% probability on the correct pair out of thousands of candidates would represent excellent performance. As training progresses, the model should push these diagonal values higher while driving off-diagonal values lower, ultimately achieving near-perfect discrimination within each batch.

Out[7]:

Visualization

Heatmap of a 3x3 image-text similarity matrix where diagonal entries near 0.9 are highlighted in dark orange and off-diagonal entries near 0.1 to 0.3 appear in light yellow. — Similarity matrix for the worked example with 3 image-text pairs. Diagonal entries (correct pairs, values near 0.85 to 0.90) appear in dark orange, while off-diagonal entries (incorrect pairs, values from 0.1 to 0.3) appear in light yellow. The contrastive loss maximizes diagonal values relative to off-diagonal values in each row and column simultaneously.

Averaging all three image-to-text losses and computing the symmetric text-to-image component gives the final batch loss. This process encourages the model to adjust embeddings so that correct pairs (diagonal values) have higher similarity than incorrect pairs (off-diagonal values), effectively learning to discriminate between matching and non-matching pairs within the batch. The gradients flow back through both encoders, updating weights to increase the separation between positive and negative pairs. Over millions of such batches, the encoders converge toward representations where visual and linguistic descriptions of the same concept occupy the same neighborhood in the shared embedding space.