Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

CLIP › Code Implementation

Michael BrenndoerferFebruary 8, 20267 min read

Language AI HandbookAll chapters

Part 50 of 67Multimodal Models

Chapter 382CLIP › Code Implementation

Reading section 7 of 11 · Code Implementation · 7 minSection 7 of 11 · 7 min

Section 7 of 11 · 7 min read

CLIP › Code ImplementationOne section · Sections in this chapter7 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's implement CLIP-based zero-shot classification using the Hugging Face Transformers library. We'll walk through loading the model, processing inputs, and performing classification.

In[8]:

Code

from transformers import CLIPModel, CLIPProcessor

# Load pre-trained CLIP model and processor
model_name = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(model_name)
processor = CLIPProcessor.from_pretrained(model_name)

# Model configuration details (uncomment to view after loading)
# print(f"Model loaded: {model_name}")
# print(f"Text encoder layers: {model.text_model.config.num_hidden_layers}")
# print(f"Image encoder layers: {model.vision_model.config.num_hidden_layers}")
# print(f"Embedding dimension: {model.config.projection_dim}")

Out[9]:

Console

Model loaded: openai/clip-vit-base-patch32
Text encoder layers: 12
Image encoder layers: 12
Embedding dimension: 512

The CLIP ViT-B/32 model uses a 12-layer vision transformer with patch size 32 and a 12-layer text transformer, both projecting to a 512-dimensional shared space. This shared dimension is the key architectural constraint that makes cross-modal comparison possible.

Now let's download a sample image and prepare candidate labels for zero-shot classification:

In[10]:

Code

import io

import requests
from PIL import Image, ImageDraw

# Download a real dog photo so CLIP can actually classify it correctly
dog_url = "https://images.unsplash.com/photo-1543466835-00a7907e9de1?w=320&auto=format"
headers = {"User-Agent": "Mozilla/5.0"}
try:
    response = requests.get(dog_url, headers=headers, timeout=15)
    response.raise_for_status()
    image = Image.open(io.BytesIO(response.content)).convert("RGB")
except requests.RequestException:
    image = Image.new("RGB", (320, 240), color=(210, 180, 130))
    draw = ImageDraw.Draw(image)
    draw.ellipse((95, 55, 225, 185), fill=(140, 95, 45))
    draw.text((118, 190), "dog", fill=(40, 30, 20))

# Define candidate classes for zero-shot classification
candidate_labels = ["a dog", "a cat", "a bird", "a car", "a tree"]
prompt_template = "a photo of a {}"

# Create full prompts using the template
text_inputs = [prompt_template.format(label) for label in candidate_labels]

print(f"Image size: {image.size}")
print(f"Candidate labels: {candidate_labels}")

Out[11]:

Console

Image size: (320, 240)
Candidate labels: ['a dog', 'a cat', 'a bird', 'a car', 'a tree']

Note the use of the prompt template. We construct "a photo of a dog" rather than just "dog" because this matches the distribution of captions in the training data and produces more stable embeddings.

Next, we process the inputs through CLIP. The processor handles tokenization for text and normalization plus reshaping for images:

In[12]:

Code

# Process inputs: tokenize text and preprocess image
inputs = processor(
    text=text_inputs, images=image, return_tensors="pt", padding=True
)

print(f"Input IDs shape: {inputs['input_ids'].shape}")
print(f"Pixel values shape: {inputs['pixel_values'].shape}")
print(f"Attention mask shape: {inputs['attention_mask'].shape}")

Out[13]:

Console

Input IDs shape: torch.Size([5, 8])
Pixel values shape: torch.Size([1, 3, 224, 224])
Attention mask shape: torch.Size([5, 8])

The processor converts our 5 text prompts into token tensors of shape (5, 77) where 77 is CLIP's maximum context length, and the image into a pixel tensor of shape (1, 3, 224, 224) representing batch size, RGB channels, and resolution. CLIP's processor normalizes the image using the mean and standard deviation of the training data, centering each channel around zero and scaling to unit variance. This preprocessing ensures the image encoder receives inputs in the same statistical regime it saw during training.

Now we compute the embeddings and similarities:

In[14]:

Code

import torch

# Forward pass through CLIP
with torch.no_grad():
    outputs = model(**inputs)

    # Extract embeddings
    image_embeds = outputs.image_embeds  # Shape: (1, 512)
    text_embeds = outputs.text_embeds  # Shape: (5, 512)

    # L2 normalization (already normalized by CLIP, but applying explicitly for clarity)
    image_embeds = image_embeds / image_embeds.norm(dim=-1, keepdim=True)
    text_embeds = text_embeds / text_embeds.norm(dim=-1, keepdim=True)

    # Compute cosine similarities (dot product of normalized vectors)
    similarities = (image_embeds @ text_embeds.T).squeeze(0)

    # Convert to probabilities using softmax with temperature scaling
    temperature = model.logit_scale.exp().item()
    probs = (similarities * temperature).softmax(dim=0)

print(f"Image embeddings shape: {image_embeds.shape}")
print(f"Text embeddings shape: {text_embeds.shape}")
print(f"Temperature (learned): {temperature:.4f}")

Out[15]:

Console

Image embeddings shape: torch.Size([1, 512])
Text embeddings shape: torch.Size([5, 512])
Temperature (learned): 100.0000

The model.logit_scale stores the learned log-temperature parameter. Taking its exponential gives the actual temperature value, typically around 100 (the inverse of 0.01 in terms of $\tau$ ). This means the model uses an extremely sharp softmax that concentrates essentially all probability mass on the highest-scoring candidate. The learned temperature is much larger than the initialization value because training on 400 million diverse examples allows the model to become very confident in its similarity judgments for semantically distinct categories. Finally, let's visualize the classification results:

Out[16]:

Visualization

Bar chart showing CLIP classification probabilities for dog, cat, bird, car, and tree classes. — Zero-shot classification results using CLIP on a dog photograph. The model correctly identifies the image as a dog with very high confidence, while assigning near-zero probability to semantically unrelated categories like car and tree. The concentration of probability on the correct class demonstrates the effectiveness of the learned embedding space and temperature scaling.

The results demonstrate CLIP's ability to classify without task-specific training. The model typically assigns probability greater than 0.95 to "a dog" for clear dog photographs, with negligible probability distributed among the other candidates. This extreme concentration reflects the large learned temperature, which sharpens the softmax so strongly that even a modest advantage in cosine similarity translates into near-certain prediction.

Embedding Extraction and Similarity SearchLink Copied

Beyond classification, CLIP embeddings enable semantic image search and clustering. Because the embedding space is shared across modalities, you can use text queries to retrieve images, or use images to retrieve similar images, all within the same vector space. Let's extract embeddings for multiple images and compute pairwise similarities:

In[17]:

Code

# Download real images of semantically distinct subjects so CLIP embeddings
# reflect semantic content rather than colour alone.
import io

import requests

image_urls = [
    (
        "dog",
        "https://images.unsplash.com/photo-1543466835-00a7907e9de1?w=160&auto=format",
    ),
    (
        "cat",
        "https://images.unsplash.com/photo-1514888286974-6c03e2ca1dba?w=160&auto=format",
    ),
    (
        "car",
        "https://images.unsplash.com/photo-1541899481282-d53bffe3c35d?w=160&auto=format",
    ),
    (
        "dog2",
        "https://images.unsplash.com/photo-1587300003388-59208cc962cb?w=160&auto=format",
    ),
]

headers = {"User-Agent": "Mozilla/5.0"}
images = []
labels = []
for label, url in image_urls:
    try:
        resp = requests.get(url, headers=headers, timeout=15)
        resp.raise_for_status()
        img = Image.open(io.BytesIO(resp.content)).convert("RGB")
    except requests.RequestException:
        img = Image.new("RGB", (160, 160), color=(220, 220, 220))
        ImageDraw.Draw(img).text((40, 70), label, fill=(30, 30, 30))
    images.append(img)
    labels.append(label)

# Process all images
image_inputs = processor(images=images, return_tensors="pt", padding=True)

# Extract embeddings
with torch.no_grad():
    vision_out = model.get_image_features(
        pixel_values=image_inputs["pixel_values"]
    )
    image_features = vision_out.pooler_output
    image_features = image_features / image_features.norm(dim=-1, keepdim=True)

# Compute similarity matrix
similarity_matrix = (image_features @ image_features.T).numpy()

print(f"Similarity matrix shape: {similarity_matrix.shape}")
print(f"Labels: {labels}")

Out[18]:

Console

Similarity matrix shape: (4, 4)
Labels: ['dog', 'cat', 'car', 'dog2']

Now let's visualize the similarity matrix to see how CLIP clusters semantically similar images:

Out[19]:

Visualization

Annotated heatmap showing runtime cosine similarities among four images labeled dog, cat, car, and dog2, with a color scale from zero to one. — Runtime cosine similarity matrix for CLIP embeddings of four images: two dogs, a cat, and a car. The diagonal shows self-similarity, while the off-diagonal cells expose the neighborhood structure for these particular samples. Comparing the dog-dog score with dog-cat, dog-car, and cat-dog scores also shows why a small example should be interpreted as a diagnostic rather than a benchmark.

The similarity matrix is best read comparatively. The diagonal is 1.0 by construction, and each off-diagonal entry shows how these particular images relate in CLIP's embedding space. The dog pair should be compared with the dog-cat and dog-car pairs, but one four-image sample is not enough to establish categorical separation: exact scores depend on image content, composition, and preprocessing. A proper evaluation would repeat this analysis across many labeled examples and test whether same-class pairs are consistently more similar than different-class pairs.