CLIP › The Contrastive Training Objective

CLIP's training objective is deceptively simple: given a batch of $N$ image-text pairs, maximize the cosine similarity of the $N$ correct pairs while minimizing the similarity of the $N^2 - N$ incorrect pairs. This is a form of instance discrimination implemented through noise contrastive estimation. The elegance of this approach lies in its scalability: it requires no manual annotation beyond the implicit supervision provided by the co-occurrence of images and text on the internet.

During training, CLIP samples a mini-batch of $N$ image-text pairs $(x_i, t_i)$ where $x_i$ is an image and $t_i$ is its corresponding text caption. The goal is to learn embeddings $f(x_i)$ and $g(t_i)$ such that the dot product $f(x_i) \cdot g(t_j)$ is large when $i = j$ and small when $i \neq j$.

The construction of batches is itself a necessary design decision. Because the contrastive objective treats all other items in the batch as negative examples, the batch size directly determines the difficulty of the learning task. Larger batches provide more negative examples, creating a more challenging discrimination task that typically results in richer, more generalizable representations. With a batch size of 256, each image must be distinguished from 255 incorrect text descriptions. With a batch size of 32,768, the task becomes dramatically harder, requiring the model to learn finer-grained distinctions to correctly identify its matching text among thousands of alternatives.

This relationship between batch size and representation quality follows an intuitive pattern: with more negatives to discriminate against, the model must learn more fine-grained features to distinguish between similar but distinct concepts. However, this also means that batch composition matters. If a batch contains multiple similar images (several different dog photos), the model must learn to distinguish between fine-grained variations. The presence of these "hard negatives," pairs that are semantically similar but not matched, provides the strongest gradient signal for learning fine-grained visual distinctions. Batches assembled purely from random sampling will produce some hard negatives naturally, but later work has explored explicit hard negative mining to accelerate learning.

The key insight about why internet data provides sufficient supervision is that the implicit assumption made during batch construction, that the only correct pairing is the one that co-occurred in the original web data, is mostly but not perfectly true. Sometimes an image is paired with a caption that loosely describes it, or a caption that describes the scene rather than the main subject. This noise in the supervision signal turns out to be tolerable at scale: with hundreds of millions of examples, the correct signal overwhelms the noise, and the model learns to align visual and linguistic representations despite imperfect labels.

CLIP employs a symmetric loss function that treats both image-to-text and text-to-image retrieval tasks equally. For a single batch, we compute an $N \times N$ similarity matrix $S$. We compute the scaled similarity score $S_{ij}$ between image $i$ and text $j$ by taking the dot product of their L2-normalized embeddings and applying a temperature scaling parameter. This scaling controls the sharpness of the probability distribution in the contrastive loss:

$$S_{ij} = \frac{f(x_i) \cdot g(t_j)}{\tau}$$

where:

• $S_{ij}$: the scaled similarity score between image $i$ and text $j$
• $f(x_i)$: the L2-normalized embedding of image $x_i$ from the image encoder
• $g(t_j)$: the L2-normalized embedding of text $t_j$ from the text encoder
• $\tau$: the temperature parameter, initialized to approximately 0.07 and learned during training (implemented as $\exp(\tau_{\text{learned}})$ to ensure positivity)

The temperature parameter $\tau$ controls how sharply the model discriminates between positive and negative pairs. A small temperature sharpens the softmax distribution, putting probability mass on the highest-similarity pair and creating strong gradient signals for hard negatives. A large temperature flattens the distribution, treating all pairs more equally. The optimal temperature depends on the difficulty of the discrimination task and the quality of the training data: noisier data benefits from a flatter distribution, while clean data with many hard negatives benefits from a sharper one. By making the temperature a learned parameter, CLIP allows the model to find the optimal value for its particular training conditions.

The temperature also prevents a degenerate solution. Without temperature scaling, the model could minimize the contrastive loss by simply making all embeddings point in the same direction (maximizing all dot products equally) or by making the magnitude of positive pairs very large while keeping negatives small. L2 normalization prevents the magnitude trick, and a learned temperature prevents the model from exploiting the sharpness of the softmax in degenerate ways. These two constraints together push the model toward a well-structured embedding space.

The loss consists of two cross-entropy terms.

Image-to-Text Loss: To align images with their corresponding texts, we treat each image as a query and all texts in the batch as potential matches. For each image $i$, we want to maximize the probability that the model assigns to the correct text $i$ compared to all other texts. This is computed using cross-entropy:

$$\mathcal{L}_{\text{i2t}} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{ii})}{\sum_{j=1}^{N} \exp(S_{ij})}$$

where:

• $\mathcal{L}_{\text{i2t}}$: the image-to-text contrastive loss
• $N$: the batch size (number of image-text pairs)
• $S_{ii}$: the similarity score between image $i$ and its matching text $i$ (diagonal elements)
• $S_{ij}$: the similarity score between image $i$ and text $j$ (possibly mismatched)
• The fraction represents the softmax probability of the correct text given image $i$

Text-to-Image Loss: Similarly, we treat each text as a query and all images as potential matches. For each text $i$, we want the correct image $i$ to have the highest similarity score compared to all other images in the batch. This computes the reverse direction:

$$\mathcal{L}_{\text{t2i}} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(S_{ii})}{\sum_{j=1}^{N} \exp(S_{ji})}$$

where:

• $\mathcal{L}_{\text{t2i}}$: the text-to-image contrastive loss
• $N$: the batch size
• $S_{ii}$: the similarity between text $i$ and its matching image $i$
• $S_{ji}$: the similarity between text $i$ and image $j$ (note the index order)
• The denominator sums over all images $j$ for a fixed text $i$, contrasting the correct pair against all other images in the batch

Notice that the denominator sums over the rows (images) rather than columns (texts). By optimizing both directions simultaneously, the model learns a symmetric alignment where images retrieve their matching texts and texts retrieve their matching images. This keeps the shared embedding space respects semantic relationships from both modalities. The total loss combines both directions symmetrically:

$$\mathcal{L}_{\text{CLIP}} = \frac{1}{2} (\mathcal{L}_{\text{i2t}} + \mathcal{L}_{\text{t2i}})$$

where:

• $\mathcal{L}_{\text{CLIP}}$: the total CLIP contrastive loss
• $\mathcal{L}_{\text{i2t}}$: the image-to-text loss component
• $\mathcal{L}_{\text{t2i}}$: the text-to-image loss component
• The factor of $\frac{1}{2}$ averages the two directional losses to ensure balanced gradients

This symmetric formulation ensures that the model learns bidirectional alignment: images should retrieve their texts, and texts should retrieve their images. The symmetry reflects the basic equivalence of the two modalities in the shared embedding space. Neither modality is the privileged "ground truth." Both are equally valid entry points into the semantic space. If you embed an image and then search for the closest text, you should find the same pair as if you started from that text and searched for the closest image. Without symmetry, the model could exploit the asymmetry by placing images and texts in disconnected regions of the space that happen to satisfy the one-directional loss without achieving true semantic alignment.

The connection between cross-entropy and contrastive learning is worth making explicit. Standard cross-entropy in a classification setting measures how well the model assigns probability to the correct class. Here, the "class" for image $i$ is text $i$, and the "logits" are the scaled similarity scores $S_{ij}$ between image $i$ and each text $j$ in the batch. Maximizing the probability of the correct text is equivalent to minimizing the negative log-likelihood of the correct pairing.

This formulation is sometimes called InfoNCE (Information Noise Contrastive Estimation). The connection to information theory is direct: the InfoNCE loss is a lower bound on mutual information between image and text representations. By minimizing this loss, the model maximizes a lower bound on the mutual information between the two modalities, encouraging the encoders to capture shared semantic content rather than modality-specific noise. This theoretical connection explains why contrastive learning produces representations that are both discriminative and generalizable, the two properties needed for zero-shot transfer.

CLIP was trained on 400 million image-text pairs collected from the internet (the WebImageText dataset, or WIT). At this scale, the contrastive objective becomes computationally efficient compared to autoregressive alternatives. While predicting text tokens pixel-by-pixel or token-by-token requires sequential generation with a loss computed over a long sequence, contrastive learning computes a single $N \times N$ matrix of dot products per batch. This parallelism enables training on massive datasets within reasonable timeframes.

The batch size becomes a necessary hyperparameter. CLIP was trained with batch sizes ranging from 32,768 to 65,536, requiring significant distributed training infrastructure. Large batches provide more negative examples, creating a harder contrastive task that produces richer representations. However, this scaling comes with computational costs, requiring distributed training across many accelerators to process these large batches within memory constraints. The $N \times N$ similarity matrix requires $O(N^2)$ memory, which means doubling the batch size quadruples the memory requirement for the similarity computation alone. This constraint drove the design of efficient distributed implementations where different batches are processed on different devices but share their computed embeddings for the similarity calculation.

An important practical consequence of large-batch training is that the model rarely sees the same training example twice. With 400 million examples and a batch size of 32,768, a single epoch requires about 12,000 gradient steps. CLIP was trained for approximately 32 epochs, meaning each example was seen roughly 32 times total during training. This data efficiency requirement puts a premium on the quality and diversity of the training data. Unlike fine-tuning scenarios where you might use thousands of augmented copies of a small dataset, CLIP's training relies on the natural diversity of internet-scraped image-text pairs.