Multimodal Foundations › A Worked Example: Batch Contrastive Loss

Let us walk through the computation of the InfoNCE loss for a small batch of three image-text pairs to make the mathematics concrete. This example contains every step with explicit numbers so that the relationship between the geometry of the embedding space and the value of the loss is clear.

Suppose after encoding and L2-normalizing, we have embeddings in $\mathbb{R}^2$ (two dimensions for visualization purposes):

• Image embeddings: $v_1 = [1, 0]$, $v_2 = [0, 1]$, $v_3 = [-1, 0]$ (all unit vectors)
• Text embeddings: $t_1 = [0.9, 0.4]$ (similar to $v_1$), $t_2 = [-0.3, 0.95]$ (similar to $v_2$), $t_3 = [-0.9, -0.4]$ (similar to $v_3$), all after L2-normalization.

where $(v_i, t_i)$ are matched pairs.

Step 1: Compute the similarity matrix $S = V T^T$.

$$S_{ij} = v_i \cdot t_j$$

For our example (using approximate values after normalizing $t_1, t_2, t_3$):

$$S = \begin{bmatrix} v_1 \cdot t_1 & v_1 \cdot t_2 & v_1 \cdot t_3 \\ v_2 \cdot t_1 & v_2 \cdot t_2 & v_2 \cdot t_3 \\ v_3 \cdot t_1 & v_3 \cdot t_2 & v_3 \cdot t_3 \end{bmatrix} \approx \begin{bmatrix} 0.90 & -0.30 & -0.90 \\ 0.44 & 0.95 & -0.44 \\ -0.90 & 0.30 & 0.90 \end{bmatrix}$$

The diagonal entries are large and positive (high similarity for matched pairs) while off-diagonal entries are smaller or negative (low similarity for mismatched pairs).

Step 2: Scale by temperature $\tau = 0.1$.

$$S' = S / \tau \approx \begin{bmatrix} 9.0 & -3.0 & -9.0 \\ 4.4 & 9.5 & -4.4 \\ -9.0 & 3.0 & 9.0 \end{bmatrix}$$

The temperature scaling amplifies the differences, making the softmax much more peaked.

Step 3: Compute row-wise softmax for the image-to-text direction.

For image $i = 1$:

$$\text{softmax}(S'_{1, :}) = \frac{[\exp(9.0), \exp(-3.0), \exp(-9.0)]}{\exp(9.0) + \exp(-3.0) + \exp(-9.0)}$$

Since $\exp(9.0) \approx 8103$ and $\exp(-3.0) \approx 0.05$ and $\exp(-9.0) \approx 0.0001$, the softmax is approximately $[0.9999, 0.0001, 0]$. Almost all probability mass sits on the matched pair (index 1).

Step 4: Compute cross-entropy loss for image 1.

$$\ell_1^{(V \to T)} = -\log p_{11} \approx -\log(0.9999) \approx 0.0001$$

Very low loss because the matched pair has very high similarity and temperature scaling made the difference extremely sharp.

Step 5: Repeat symmetrically for text-to-image. The structure is identical by the construction of our example.

Step 6: Average all terms to get the final loss. In this idealized example, after several steps of training that pushed embeddings toward high similarity for matched pairs, the loss is very near zero.

Now contrast this with an untrained model where embeddings are random. Random unit vectors in high dimensions have expected cosine similarity near zero, so all entries of $S$ are near zero and the softmax distribution is nearly uniform. In that case, $p_{ii} \approx 1/N$ and the loss is approximately $\log N$, which for a batch of 32,768 is about 10.4. Training drives the loss from $\log N$ toward zero.