CLIP › Zero-Shot Transfer

CLIP enables zero-shot classification without any task-specific training. Traditional supervised learning requires training data for every category of interest. CLIP eliminates this requirement by letting you to describe categories through natural language. This capability turns the model from a fixed classifier into a flexible vision system that can be directed through language, much like how you can instruct a person to identify new objects simply by describing them. You do not need to train a person by showing them 10,000 labeled examples before they can recognize a new animal species; describing the animal in words is enough. CLIP extends this natural human capability to neural networks.

The zero-shot capability arises from two converging properties of the trained model. First, the shared embedding space encodes semantic similarity across modalities, so images and their descriptions map to nearby points. Second, the space is continuous and generalizes across concepts rather than discretizing into fixed categories. When you embed the text "a photo of a trebuchet," the resulting vector lands in a region of the embedding space associated with medieval siege weaponry, large wooden structures, and counterweight mechanisms. Any image that shares these visual properties will also land near that region, even if the model never encountered a trebuchet during training. The embedding space does not have explicit "trebuchet slot"; instead, trebuchet-like images occupy a region defined by their visual similarities to other things the model did see.

To classify an image without task-specific training:

1. Construct prompt templates: Instead of using bare class names like "tench" or "red panda," CLIP uses templates like "a photo of a {label}" or "a rendering of a {label}"
2. Compute text embeddings: For each candidate class, embed the templated description using the text encoder
3. Compute image embedding: Pass the target image through the image encoder
4. Calculate similarities: Compute cosine similarity between the image embedding and all text embeddings
5. Predict: Select the class with the highest similarity score

This pipeline effectively turns classification into a retrieval problem. Rather than learning fixed decision boundaries during training, the model dynamically computes the similarity between the input image and arbitrary text descriptions provided at inference time. This flexibility allows the same model to perform classification on entirely new datasets without any parameter updates, simply by changing the set of text prompts.

The computational cost of zero-shot classification is minimal for small class sets. For each classification, you need one forward pass through the image encoder and $K$ forward passes through the text encoder (one per candidate class), followed by $K$ dot products. For a dataset with 1,000 classes, you can precompute all 1,000 text embeddings once and reuse them across all test images. The marginal cost per image is then just one image encoder forward pass plus 1,000 dot products, making the system computationally competitive with standard classifiers despite requiring no training on the target dataset.

CLIP's performance depends heavily on prompt construction. The original paper found that using prompt templates significantly outperformed bare class names. For example, "a photo of a dog" works better than simply "dog" because it matches the distribution of training captions, which tend to be full sentences describing images. This phenomenon highlights an important aspect of CLIP's learning: the model was trained on natural language descriptions, not isolated category labels, so it performs best when queries match that training distribution. When you query with a bare label, the text encoder processes it as a short, context-free fragment, which may land in a different part of the embedding space than the descriptions that appeared alongside images during training.

The sensitivity to prompts reveals something deeper about how the model works. The text encoder is not simply extracting keywords; it is building a representation sensitive to the full linguistic context. "A photo of a dog" activates visual associations tied to photographic images of dogs, while "a sketch of a dog" would activate different visual features. This context-sensitivity is precisely what enables zero-shot transfer: by choosing your prompt carefully, you can direct the model's attention toward specific visual properties of the target concept.

The authors recommend ensemble averaging over multiple prompt templates:

It is worth understanding where zero-shot CLIP sits relative to other evaluation regimes. A linear probe evaluation trains a linear classifier on top of frozen CLIP features for the specific target dataset. This requires labeled data but does not update the backbone. Full fine-tuning updates all parameters with labeled data. Zero-shot requires no labeled data at all.

CLIP's zero-shot performance on ImageNet is approximately 76% top-1 accuracy with the ViT-L/14 model, competitive with a supervised ResNet-50 (76.1%) trained on the full ImageNet training set with millions of labeled examples. This comparison is striking: CLIP achieves comparable accuracy to a model that was given 1.2 million labeled training examples from exactly that test distribution, while receiving no examples at all. On many other benchmarks, linear probe CLIP significantly outperforms zero-shot CLIP, suggesting that while the representations are rich, the mapping from features to class-specific predictions benefits from at least minimal supervision.

The performance gap between zero-shot and linear probe evaluations reveals an important limitation of zero-shot classification: the text prompt must describe each class in a way that aligns with how that class appeared in training data. For fine-grained classification benchmarks like the Oxford Pets (distinguishing 37 cat and dog breeds) or EuroSAT (satellite imagery land use), zero-shot CLIP performs significantly below the linear probe variant because the text descriptions of fine-grained categories do not capture the specific visual differences the model needs to discriminate. A linear probe learns exactly which features distinguish "Siamese cat" from "Maine Coon" from the training data, which text prompts cannot replicate without that information.