Multimodal Foundations › Code Implementation
Language AI HandbookAll chapters Multimodal Foundations › Code ImplementationOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us implement a minimal multimodal contrastive learning system from scratch to build concrete intuition for how joint embedding spaces are trained.
Setup and Data Preparation
We start by creating synthetic image and text embeddings to simulate what pretrained encoders would produce.
import numpy as np
import torch
# Simulate paired data: 5 semantic categories
# In practice these come from real image-text pairs
n_pairs = 200
n_categories = 5
d_raw = 64 # raw feature dimension
d_embed = 32 # joint embedding dimension
# Create cluster centers for each semantic category
centers = torch.randn(n_categories, d_raw)
# Assign each pair a category
labels = torch.randint(0, n_categories, (n_pairs,))
# Give each pair a shared instance-level signal within its semantic category.
# This lets the contrastive objective learn both broad categories and exact matches.
pair_latents = centers[labels] + 0.35 * torch.randn(n_pairs, d_raw)
# Generate image features: shared latent content + modality-specific noise
image_features = pair_latents + 0.15 * torch.randn(n_pairs, d_raw)
# Generate text features: same category center + different noise
# Text features start in a rotated subspace to simulate modality gap
rotation = torch.randn(d_raw, d_raw)
text_features = pair_latents @ rotation.T / np.sqrt(d_raw) + 0.15 * torch.randn(
n_pairs, d_raw
)Image features shape: torch.Size([200, 64]) Text features shape: torch.Size([200, 64]) Number of semantic categories: 5 Pairs per category: 40 (average)
The synthetic data mimics real multimodal data: image and text features live in different subspaces (the rotation matrix simulates the modality gap) but share both semantic-category structure and a weaker pair-specific signal. The rotation ensures the raw features are not directly comparable, requiring the model to learn to align them.
Dual-Encoder Model
We implement a simple dual-encoder with separate projection heads for each modality.
class ModalityProjection(nn.Module):
"""Projects raw modality features into the joint embedding space."""
def __init__(self, in_dim: int, out_dim: int):
super().__init__()
self.net = nn.Sequential(
nn.Linear(in_dim, out_dim * 2),
nn.ReLU(),
nn.Linear(out_dim * 2, out_dim),
)
def forward(self, x: torch.Tensor) -> torch.Tensor:
z = self.net(x)
# L2-normalize to unit sphere for cosine similarity
return F.normalize(z, dim=-1)
class DualEncoder(nn.Module):
"""Dual-encoder with separate visual and text projection heads."""
def __init__(self, in_dim: int, embed_dim: int):
super().__init__()
self.visual_proj = ModalityProjection(in_dim, embed_dim)
self.text_proj = ModalityProjection(in_dim, embed_dim)
self.log_temperature = nn.Parameter(torch.zeros(1))
@property
def temperature(self) -> torch.Tensor:
# Clamp temperature to [0.01, 10] range for stability
return self.log_temperature.exp().clamp(0.01, 10.0)
def forward(self, image_feats: torch.Tensor, text_feats: torch.Tensor):
v = self.visual_proj(image_feats)
t = self.text_proj(text_feats)
return v, t
model = DualEncoder(in_dim=d_raw, embed_dim=d_embed)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
# Save pre-training embeddings to compare distributions before and after training
with torch.no_grad():
v_init, t_init = model(image_features, text_features)
sim_init = torch.matmul(v_init, t_init.T)
pos_sims_init = sim_init.diagonal().numpy()
neg_sims_init = sim_init[~torch.eye(n_pairs, dtype=torch.bool)].numpy()The ModalityProjection module acts as a bridge between each modality's raw feature space and the shared joint embedding space. The two-layer MLP with a ReLU nonlinearity gives the projections enough capacity to learn nonlinear transformations. The L2-normalization at the end ensures all output embeddings lie on the unit hypersphere, which is the standard for contrastive learning.
Contrastive Loss and Training
def contrastive_loss(
v: torch.Tensor, t: torch.Tensor, temperature: torch.Tensor
) -> torch.Tensor:
"""Symmetric InfoNCE contrastive loss."""
N = v.shape[0]
# Pairwise cosine similarity matrix (N x N)
# v and t are already L2-normalized
sim = torch.matmul(v, t.T) / temperature
# Targets: diagonal indices (matched pairs)
targets = torch.arange(N)
# Image-to-text and text-to-image cross-entropy
loss_v2t = F.cross_entropy(sim, targets)
loss_t2v = F.cross_entropy(sim.T, targets)
return (loss_v2t + loss_t2v) / 2
# Training loop
n_epochs = 100
batch_size = 64
losses = []
for epoch in range(n_epochs):
# Shuffle and batch
perm = torch.randperm(n_pairs)
epoch_losses = []
for start in range(0, n_pairs, batch_size):
idx = perm[start : start + batch_size]
img_batch = image_features[idx]
txt_batch = text_features[idx]
optimizer.zero_grad()
v, t = model(img_batch, txt_batch)
loss = contrastive_loss(v, t, model.temperature)
loss.backward()
optimizer.step()
epoch_losses.append(loss.item())
losses.append(np.mean(epoch_losses))Initial loss: 3.5004 Final loss: 2.6214 Reduction: 25.1% Final temperature: 0.6577
The loss decreases substantially over training, showing the encoders have learned to map matched image-text pairs to similar regions of the joint embedding space. The learned temperature reflects how sharply the model discriminates between positives and negatives: a temperature near 0.1 indicates the model has learned to produce high similarity for matched pairs and near-zero or negative similarity for mismatched pairs.
Evaluating Retrieval Quality
A direct measure of alignment quality is retrieval accuracy: given each image embedding, can we retrieve its matched text from the full pool of all texts?
def retrieval_accuracy(
model: DualEncoder, image_feats: torch.Tensor, text_feats: torch.Tensor
) -> dict:
"""Compute image-to-text and text-to-image top-k retrieval accuracy."""
with torch.no_grad():
v, t = model(image_feats, text_feats)
# Full similarity matrix (N x N)
sim = torch.matmul(v, t.T)
N = sim.shape[0]
targets = torch.arange(N)
results = {}
for k in [1, 5, 10]:
# Image-to-text: for each image, is the matched text in top-k?
top_k_v2t = sim.topk(k, dim=1).indices
hit_v2t = (
(top_k_v2t == targets.unsqueeze(1)).any(dim=1).float().mean().item()
)
# Text-to-image: for each text, is the matched image in top-k?
top_k_t2v = sim.T.topk(k, dim=1).indices
hit_t2v = (
(top_k_t2v == targets.unsqueeze(1)).any(dim=1).float().mean().item()
)
results[f"R@{k}"] = {"I2T": hit_v2t, "T2I": hit_t2v}
return results
# Evaluate on full dataset
retrieval_results = retrieval_accuracy(model, image_features, text_features)Retrieval Accuracy (R@k): Metric Image-to-Text Text-to-Image -------------------------------------------- R@1 79.0% 76.0% R@5 95.5% 95.5% R@10 99.0% 98.0%
Recall at (R@) measures the fraction of queries for which the correct match appears in the top- retrieved results. High R@1 indicates tight alignment: the correct match is the single most similar item. High R@5 or R@10 indicates the model is directionally correct but may not achieve perfect precision at rank 1. In practice on large-scale datasets like MS-COCO, state-of-the-art models achieve R@1 image-to-text retrieval above 80%.
Visualizing the Joint Embedding Space
The most intuitive way to understand what the joint embedding space has learned is to project it to two dimensions and examine the cluster structure.
Circles represent image embeddings and triangles represent text embeddings. Points of the same color belong to the same semantic category. When contrastive training succeeds, circles and triangles of the same color cluster near each other, showing that the image and text encoders have learned to map semantically equivalent content to the same region of the joint space. The five-category structure should be visible as five distinct clusters, with image and text embeddings from each category interleaved rather than separated.
Visualizing the Similarity Matrix
The diagonal of this heatmap represents matched pairs: each image compared with its corresponding text. The amber outlines make those exact pairs explicit. Because the synthetic dataset also has five shared semantic categories, some off-diagonal cells are dark as well: these are unmatched examples that remain semantically related. The model therefore learns both exact-pair alignment and broader category neighborhoods, which explains why retrieval rank remains a stricter test than category separation alone.
Training Loss Curve
Positive vs. Negative Similarity Distributions
A key diagnostic for contrastive learning is the distribution of cosine similarities for positive (matched) and negative (unmatched) pairs. Good alignment produces a clear separation between these two distributions.
Before training, positive and negative pair similarities are indistinguishable: the encoders have not yet learned what makes an image-text pair match. After training, positive pairs cluster near 1.0 and most negatives remain much lower. The smaller negative peak near 1.0 contains semantically related examples from the same synthetic category. This is a useful geometric signature: the space captures broad meaning while the exact ordering within a semantic neighborhood determines downstream retrieval quality.