Multimodal Foundations › Multimodal Pretraining at Scale

The code example above uses small synthetic data to illustrate the mechanism. Real multimodal pretraining operates at very different scales, and the scaling decisions strongly affect model quality.

CLIP was trained on 400 million image-text pairs assembled from the web. ALIGN used 1.8 billion pairs. These massive datasets are necessary because the model must learn a joint embedding space that covers the enormous diversity of visual and linguistic concepts in the world. With small datasets, the model only learns to align the specific visual and textual patterns present in training and fails to generalize to novel content.

Dataset diversity is as important as scale. A dataset of 400 million similar images, such as product photography from a single retailer, would produce much weaker representations than 400 million diverse images spanning art, nature, sports, scientific imagery, and everyday scenes. Diversity in training data leads to generalization across the broad semantic space the model must cover.

The tradeoff between scale and quality is not settled. LAION-5B collected five billion image-text pairs through aggressive web scraping with minimal filtering, while DataComp curated datasets of varying sizes with different quality thresholds. Research on DataComp showed that smaller, carefully curated datasets can match or exceed the quality of much larger noisily collected datasets on downstream benchmarks. The lesson: scale matters, but quality at scale matters more.

Contrastive learning particularly benefits from large batch sizes because larger batches contain more negative pairs. With a batch of 32,768 pairs, each image competes against 32,767 other texts during training, forcing the model to learn very precise discrimination. A batch of 64 pairs, by contrast, provides only 63 negatives per query, which is usually insufficient for learning fine-grained semantics.

CLIP's batch size of 32,768 was enabled by distributing training across hundreds of GPUs and aggregating embeddings across all devices before computing the loss. In distributed contrastive training, each device computes embeddings for its local batch shard, then all-gathers the embeddings across all devices to construct the global similarity matrix. This global-gather pattern makes the effective batch size for the loss computation equal to the total batch size across all devices rather than the local shard size.

The computational cost of this global-gather scales linearly with the number of devices, but the resulting large batch dramatically improves learning efficiency. The CLIP training recipe is fundamentally bottlenecked by the batch size: more pairs means more negatives, means sharper discrimination, means better alignment.

CLIP experiments show that visual encoders ranging from ResNet-50 (25 million parameters) to ViT-L/14 (307 million parameters) consistently benefit from scale, with larger encoders creating higher-quality embeddings on zero-shot classification and retrieval benchmarks. Text encoders of comparable scale are used alongside the visual encoders.

The choice of visual encoder architecture also matters. ResNets apply convolutional filters in a translationally equivariant way, which is well-suited to natural image statistics where objects can appear at any spatial position. ViTs divide the image into patches and process them with self-attention, which is more flexible but requires more data to learn good representations from scratch. At sufficient scale, ViTs typically outperform ResNets, which is why the largest CLIP models use ViT encoders.