Multimodal Foundations › Limitations of the Multimodal Approach

Multimodal learning has produced remarkable capabilities, but it comes with basic limitations that researchers are actively working to address.

Even after contrastive training, image and text embeddings tend to occupy different regions of the joint embedding space rather than fully intermixing. This is the modality gap. Research shows that image embeddings cluster in a tight cone in one part of the embedding hypersphere, and text embeddings cluster in a different cone. Matched pairs have their cones adjacent, but within each cone the two modalities do not interleave. The gap arises because the two encoders develop different geometric organization internally, and the contrastive objective only requires relative ordering (matched pairs closer than mismatched pairs) rather than exact positional overlap.

The modality gap has practical consequences for applications that assume images and texts share the same geometric manifold. Composing visual and textual concepts via vector arithmetic may produce vectors that fall outside either cone. Techniques like SigLIP propose modifications to the contrastive loss that reduce the gap by encouraging images and texts to occupy the same region of space, but the gap has not been fully eliminated in current systems.

Contrastive models trained on whole-image, whole-caption pairs learn global correspondences but struggle with fine-grained compositional understanding. The captions "a red ball to the left of a blue cube" and "a blue ball to the left of a red cube" may receive very similar scores from a CLIP model because the model attends to the presence of red, blue, ball, and cube without correctly reasoning about the spatial relations between them.

This limitation is well-documented by benchmarks like Winoground and ARO (Attribution, Relation, Order), which test compositional understanding by asking models to distinguish between captions that share the same words but differ in their grammatical structure and meaning. Current contrastive models perform surprisingly poorly on these benchmarks, sometimes approaching chance level, even while achieving excellent performance on standard image classification and retrieval.

The failure stems from how the training objective and the global embedding approach interact. Because the loss only cares about whether the correct image-text pair is ranked highest, the model can learn to match correct pairs without learning the compositional structure that distinguishes them from similar-but-wrong pairs. Dense alignment methods, which learn patch-level alignments between visual regions and textual phrases rather than global image-caption correspondences, partially address this by requiring the model to establish finer-grained correspondences.

Web-scraped image-text pairs reflect the biases of the web: overrepresentation of certain demographics, cultural perspectives, and domains, and underrepresentation of others. Models trained on such data inherit these biases and can exhibit harmful associations. CLIP has been shown to exhibit association biases which reflects social stereotypes present in web images and text, including associations between demographic groups and occupational roles that do not reflect actual proportions.

Curating higher-quality data and debiasing training data are active areas of research, but there is no complete solution. The scale required for good multimodal models, hundreds of millions to billions of pairs, makes manual curation impractical. Automated filtering can remove the most obvious low-quality pairs, but subtle biases are much harder to detect and remove at scale. Users of pretrained multimodal models should be aware of these limitations when deploying in sensitive applications and evaluate for bias on their specific use cases.

Standard contrastive models produce a single embedding for an entire image. They cannot answer questions about specific objects or regions, localize referred objects, or perform fine-grained visual question answering. Extending multimodal models to support grounding, the ability to connect textual references to specific image regions, requires architectural modifications beyond the dual-encoder.

Grounding requires the model to produce region-level or patch-level representations that can be associated with specific noun phrases or entity mentions. Models like OFA, Grounding DINO, and region-level contrastive approaches address this by training with region-text pairs or bounding box supervision alongside image-level pairs. These additional supervision signals force the model to learn spatial correspondences that pure global contrastive training cannot provide.

Extending joint embeddings to video introduces challenges that static image-text models cannot handle. A video is a temporal sequence of frames, and many captions describe temporal events: "a person picks up a ball and throws it." Correct alignment requires the model to understand the ordering and causality of events rather than merely detecting the presence of visual concepts.

Static contrastive models trained on video frames or clips can match global descriptions to global clips, but they struggle with temporal ordering and causal relationships. Video-specific architectures like VideoCLIP and TimeSformer introduce temporal attention that processes frames in sequence rather than treating them as a bag of independent images. Extending this to event-level temporal alignment, where specific parts of a video correspond to specific parts of a description, remains an open research problem.