CLIP › Architecture

CLIP consists of two parallel encoders that process different input modalities but produce embeddings in the same dimensional space. This architectural choice reflects a deliberate design philosophy: each modality requires specialized processing to handle its unique structure, but the final representations must live in a common geometric space where direct comparison is possible. The parallel structure allows for efficient inference because images and text can be encoded independently and then compared via simple vector operations.

The independence of the two encoders is both a strength and a constraint. Because each encoder processes its modality separately, you can precompute embeddings for large image databases offline, then quickly find matching text queries by comparing vectors without re-running the image encoder. This property makes CLIP extremely practical for retrieval applications. The constraint is that the two encoders cannot attend to each other during encoding: the image encoder does not know what text will be compared against it, and the text encoder does not know what image it will be matched with. All cross-modal reasoning must happen implicitly through the shared embedding space rather than through direct attention across modalities.

The image encoder turns visual input into fixed-dimensional vectors. CLIP was trained with two possible backbone architectures, which reflects the state of computer vision at the time of development:

• ResNet-based: A modified ResNet-50 or ResNet-101 with attention pooling instead of global average pooling, along with several other improvements to the base architecture including a wider final layer and a different pooling mechanism.
• Vision Transformer (ViT): As discussed in the previous chapter on Vision Transformers, CLIP uses ViT-B/32, ViT-B/16, and ViT-L/14 variants, where the numbers indicate the patch size in pixels.

The choice between convolutional and transformer architectures involves trade-offs between computational efficiency, inductive biases, and representational capacity. Convolutional networks bring strong priors about local spatial structure and locality as well as translation invariance. These priors are appropriate for natural images because nearby pixels tend to be semantically related, and the same visual pattern (an eye, a wheel) should be recognized regardless of where it appears in the image. Transformers lack these priors by design, instead learning all spatial relationships from data. With sufficient training data and compute, this flexibility allows transformers to discover image structure that convolutional networks might not find because their architecture directs attention toward local features. The ViT variants generally produce better embeddings for CLIP because the self-attention mechanism can capture global contextual relationships that are important for matching images to text descriptions.

The ViT variant follows the standard architecture described in the previous chapter: an image is divided into fixed-size patches, linearly embedded, augmented with position embeddings, and processed through transformer blocks. The final representation is taken from the [CLS] token (or average pooled across all patch tokens), then projected via a linear layer to the multimodal embedding dimension, typically 512 or 768 dimensions depending on the model variant. This projection step is important because it allows the vision encoder to maintain its native output dimensionality during pre-training while adapting to the shared multimodal space where cross-modal comparison occurs. Think of it as a learned translation layer that converts the visual encoder's native language into the shared language of the joint embedding space.

The patch size choice involves a direct speed-accuracy trade-off. ViT-B/32 splits a 224x224 image into $7 \times 7 = 49$ patches of size 32x32 pixels. ViT-B/16 uses 16x16 patches, creating $14 \times 14 = 196$ patches from the same image. Smaller patches capture finer visual detail but require quadratically more computation in the attention layers. ViT-L/14 represents the best-performing configuration in the original paper, using 14x14 patches in a larger (L) model capacity, at significantly higher computational cost. The patch size directly affects which visual details the model can represent and which it must average away.

The text encoder is a transformer decoder stack with masked self-attention, structurally similar to the GPT architecture described in Part XXVIII. The use of a decoder-only architecture, rather than an encoder (BERT-style) design, is a somewhat surprising choice for a retrieval system where you might expect a bidirectional encoder to produce richer representations. The authors found that the GPT-style decoder produced competitive or better representations for this task, possibly because the causal attention mechanism, despite attending only to left context within each token, still produces a final [EOS] representation that aggregates information from the full sequence.

Key specifications for the text encoder are:

• Base model: 63 million parameters with 12 layers, 512 hidden width, and 8 attention heads
• Larger variant: 82 million parameters with 12 layers, 768 hidden width, and 12 attention heads
• Context length: 76 tokens, shorter than typical language models to match the typical length of image captions
• Vocabulary: Byte Pair Encoding (BPE) with approximately 49,000 tokens

The relatively short context length of 76 tokens reflects an important characteristic of the training data. Image captions found on the internet tend to be brief descriptions, not multi-paragraph articles. By limiting the context length, CLIP reduces computational requirements while capturing the needed descriptive content needed for visual-semantic alignment. In practice, very few image captions exceed 76 tokens, so this constraint rarely limits the model's ability to process training examples. For zero-shot classification, the prompts used are even shorter, typically under 20 tokens.

The text encoder processes tokenized captions and produces a sequence of hidden states. Similar to the image encoder, the final [EOS] token representation is extracted and projected to the shared embedding space via a linear transformation. The [EOS] token is a summary token, aggregating information from the entire caption before the projection. Because the decoder uses causal masking, each token can only attend to previous tokens, but by the time the model processes the final [EOS] token, it has attended to all preceding content. This makes the [EOS] representation a natural aggregation point for the full caption meaning.

An important detail is that both encoders use separate projection matrices to map into the shared space. These projections are learned jointly with the encoders during contrastive training. The projection for the image encoder might have a very different structure than the projection for the text encoder. This reflects the different geometric properties of visual versus linguistic representations before alignment. The joint training ensures that both projections learn to map into a space where cosine similarity corresponds to semantic similarity across modalities.

Both encoders output embeddings that undergo two necessary transformations before entering the shared multimodal space:

1. Linear projection: A learned matrix maps encoder outputs to the shared multimodal dimension
2. L2 normalization: Embeddings are scaled to unit length, so that similarity calculations depend only on direction, not magnitude

The linear projection layers serve as adaptation mechanisms that allow the pre-trained encoders to align their native representations with the common embedding space. Without these projections, the image and text representations might occupy different subspaces or have incompatible scales, preventing effective cross-modal comparison. You can think of the projections as coordinate system transformations: both encoders produce their representations in their own native coordinates, and the projections rotate and scale these representations until they live in a common coordinate system where semantic similarity corresponds to geometric proximity.

L2 normalization is deceptively important. By constraining all embeddings to the unit hypersphere, the dot product between any two embeddings equals their cosine similarity, a value between -1 and 1 that measures the angle between the vectors. This normalization removes the magnitude component entirely, so the model cannot exploit the contrastive objective by simply making positive pairs larger in magnitude while keeping the angle unchanged. Every embedding carries exactly the same total "energy," forcing the model to encode all information through the direction of the vector rather than its length. This geometric constraint simplifies the loss surface and produces more uniform, well-calibrated representations.