Multimodal Foundations › Why Is Multimodal Learning Hard?

Before discussing solutions, it is worth examining why multimodal learning is difficult. The challenges extend beyond engineering hurdles. They reflect deep structural differences between modalities that arise from how information is captured and organized in the physical world.

Different modalities live in incompatible native spaces. A raw image is a tensor of shape $[H \times W \times 3]$, typically containing values between 0 and 255. A raw text sequence is a list of integer token IDs drawn from a vocabulary of 30,000 to 100,000 elements. These live in completely different mathematical spaces with different dimensionalities, data distributions, and inductive biases. Simply concatenating pixel values with token IDs would produce meaningless noise: the model would have no way to discover that the visual content of a patch and the meaning of a word are related to each other.

Bridging this gap requires learning modality-specific encoders that project each modality into a common vector space, where distances and directions carry consistent meaning across modalities. This is the core technical challenge of multimodal learning: the encoders must independently extract high-level features from each modality before those features can be compared in the shared space.

The representation gap is also why pretraining matters so much in multimodal systems. A randomly initialized image encoder that produces 768-dimensional vectors and a randomly initialized text encoder that also produces 768-dimensional vectors are not creating comparable representations just because they share the same output dimensionality. The dimensions are not aligned: dimension 42 in the image encoder might respond to edge features while dimension 42 in the text encoder might respond to verb tense. Pretraining each encoder on large unimodal data before multimodal training ensures that both encoders start from rich, semantically organized representations.

Even when two modalities convey the same information, their signals are not synchronized in an obvious way. Consider a cooking video with narration. At any given moment, the speaker might say "now add the garlic" while the visual shows the chef reaching for the garlic. The semantic content is aligned, but the precise correspondence between visual frames and audio segments varies over time and depends on context. A few hundred milliseconds of lag between speech and action is normal and unproblematic for a human viewer but can confuse a model trying to learn temporal correspondences.

Text-image pairs present a different but related problem. The caption "a cat sitting on a red sofa" corresponds to the entire image, not to individual patches. But different patches carry different amounts of information relevant to the caption: the cat region matters most, the sofa matters somewhat, and the background barely matters at all. Learning which visual regions to align with which textual concepts is the fine-grained alignment problem, and most large-scale contrastive models only partially solve it by training on whole image-caption pairs.

The alignment problem also manifests in training data quality. Alt-text written by web authors to describe images varies enormously in quality and specificity. Some alt-text says "a dog jumping over a fence in a park," which precisely describes the image. Some says "image_1234.jpg," which is metadata with no visual correspondence. Some says "our dog, Charlie, at his favorite park," which conveys sentiment and personal context but little about the visual content. Models must extract useful alignment signal from this noisy mixture.

Raw signals from different modalities carry semantic information at very different levels of abstraction. A pixel value of 127 in the red channel is a very low-level measurement. The word "crimson" is a high-level semantic concept. To align "red" with pixels that depict red objects, the visual encoder must have already learned to group pixels by color, identify regions, separate objects from backgrounds, and understand that redness is a property of objects rather than pixels in isolation. This requires learning rich visual semantics before any meaningful cross-modal alignment can happen.

The semantic gap is why training multimodal models from scratch on raw pixels and raw text is much harder than composing pretrained unimodal encoders. A large pretrained visual encoder like ResNet or ViT has already learned to extract semantically meaningful features: activations in late layers respond to concepts like "dog," "outdoor scene," or "circular object" rather than raw pixel statistics. Starting multimodal training from these rich features dramatically narrows the semantic gap because the visual representations are already operating at a level of abstraction comparable to language.

This insight explains why the CLIP architecture, which we study in the next chapter, uses large pretrained encoders as its starting point rather than training from scratch. The encoder pretraining is where semantic richness is built; multimodal training then teaches the encoders to align their representations across modalities.

Paired multimodal data is harder to collect than unimodal data. Building a large image-text dataset by scraping alt-text from the web produces pairs with highly variable quality. Many captions describe images poorly, omit key visual content, or convey metadata about the image rather than its visual content. Audio transcripts from automatic speech recognition contain word errors and omit prosodic information. Video subtitles are often temporally misaligned with the speech they transcribe.

This noise fundamentally affects what the model can learn. If 20% of training pairs have captions that are semantically unrelated to the image, training on those pairs produces contradictory gradient signals. The model learns to associate visual features with captions that do not describe them, degrading the quality of the joint embedding space. Filtering and cleaning multimodal training data is therefore a necessary part of the training pipeline for any serious multimodal model.

Multimodal models must tolerate noisy correspondence and learn useful representations even when the pairing between modalities is imperfect. Contrastive training is particularly well-suited to this: when the noise rate is not too high, the correct positive pairs will on average receive stronger gradient signal than the random noise pairs, and the model learns despite the noise. At high noise rates, however, this tolerance breaks down.