Multimodal Foundations: Cross-Modal Alignment

Language models, as covered throughout this handbook, are remarkable at reasoning over text. They predict the next token, encode semantic relationships, and generate fluent prose. But the world is not text-only. A doctor reading an X-ray does not translate the image into words before diagnosing. A shopper evaluating a product does not transcribe a written description before deciding. Intelligence in the real world integrates signals from multiple sources simultaneously: vision, language, audio, structured data, and more.

Multimodal AI is the field that asks: how do we build systems that reason over multiple modalities at once? The question sounds natural, but the engineering and mathematical challenges run surprisingly deep. The very things that make each modality powerful in isolation, the spatial structure of images, the temporal dynamics of audio, the discrete compositional structure of text, also make them fundamentally incompatible at the raw representation level. No single architecture trivially handles all of them.

This chapter lays the conceptual and technical foundations you need to understand modern multimodal models. We start from first principles about what modalities are and why combining them is difficult, work through the mathematics of joint embedding spaces and cross-modal alignment, examine the dominant training objectives and architectural patterns, and close with a careful look at the limitations that still constrain current systems.

By the end of this chapter, you will understand why multimodality is difficult and what mathematical machinery supports it. You will also see how familiar ideas from attention and transformer-based representation learning extend into the multimodal setting. The following chapters in this part of the handbook examine CLIP, visual instruction tuning, and full vision-language architectures in detail; everything here prepares you to read them clearly.

A modality is a distinct channel of information with its own structure and statistical properties, represented in a form suited to that channel. The most common modalities in AI research today are:

• Language: Sequences of discrete tokens drawn from a finite vocabulary. You have studied language models extensively throughout this handbook. The discrete token structure, the long-range compositional dependencies, and the absence of spatial layout distinguish text from all other modalities.
• Images: 2D grids of pixel intensities, typically stored as RGB triples. Images have rich spatial structure: nearby pixels tend to be correlated, objects occupy contiguous regions, and scale-invariant features recur across spatial positions. This spatial regularity is why convolutional networks and vision transformers, which we discuss below, were developed specifically for image processing.
• Audio: 1D waveforms sampled at high frequency (commonly 16,000 or 44,100 samples per second), or equivalently 2D spectrograms representing energy across frequency and time. Audio has strong short-range temporal correlations and multi-scale structure from phonemes up through prosody and melody.
• Video: Sequences of image frames at a fixed frame rate, combining spatial structure within each frame with temporal dynamics across frames. Video is the most demanding modality in terms of data volume and alignment complexity.
• Structured data: Tabular records, knowledge graphs, or relational databases with explicit structure and typed attributes. Unlike the other modalities, structured data has no natural sequential or spatial organization.

Each modality has its own natural representation. Images have pixels arranged in a spatial grid where nearby pixels tend to be correlated. Audio has temporal correlations across short windows. Text has long-range dependencies captured by attention. These structural differences mean there is no single obviously correct way to encode all modalities into a common format, and no single neural architecture that is optimal for all of them.

The importance of distinguishing modalities is not just taxonomic. Different modalities require different preprocessing pipelines, different encoder architectures, and different inductive biases. An architecture that works well for images, such as a ViT that treats the image as a grid of patches, would be a poor fit for audio without significant modifications. Audio requires careful handling of temporal resolution and frequency, and spectrograms have very different statistics from natural image patches.

Beyond engineering differences, different modalities also carry information at different semantic granularities. A pixel is an extremely low-level unit: its value alone carries almost no meaning without context from surrounding pixels. A word token, by contrast, already carries rich semantic information: the word "dog" activates a dense set of associations before any surrounding context is processed. This disparity in semantic granularity creates basic challenges when trying to learn correspondences between modalities, and we return to it throughout the chapter.

Before discussing solutions, it is worth examining why multimodal learning is difficult. The challenges extend beyond engineering hurdles. They reflect deep structural differences between modalities that arise from how information is captured and organized in the physical world.

Different modalities live in incompatible native spaces. A raw image is a tensor of shape $[H \times W \times 3]$, typically containing values between 0 and 255. A raw text sequence is a list of integer token IDs drawn from a vocabulary of 30,000 to 100,000 elements. These live in completely different mathematical spaces with different dimensionalities, data distributions, and inductive biases. Simply concatenating pixel values with token IDs would produce meaningless noise: the model would have no way to discover that the visual content of a patch and the meaning of a word are related to each other.

Bridging this gap requires learning modality-specific encoders that project each modality into a common vector space, where distances and directions carry consistent meaning across modalities. This is the core technical challenge of multimodal learning: the encoders must independently extract high-level features from each modality before those features can be compared in the shared space.

The representation gap is also why pretraining matters so much in multimodal systems. A randomly initialized image encoder that produces 768-dimensional vectors and a randomly initialized text encoder that also produces 768-dimensional vectors are not creating comparable representations just because they share the same output dimensionality. The dimensions are not aligned: dimension 42 in the image encoder might respond to edge features while dimension 42 in the text encoder might respond to verb tense. Pretraining each encoder on large unimodal data before multimodal training ensures that both encoders start from rich, semantically organized representations.

Even when two modalities convey the same information, their signals are not synchronized in an obvious way. Consider a cooking video with narration. At any given moment, the speaker might say "now add the garlic" while the visual shows the chef reaching for the garlic. The semantic content is aligned, but the precise correspondence between visual frames and audio segments varies over time and depends on context. A few hundred milliseconds of lag between speech and action is normal and unproblematic for a human viewer but can confuse a model trying to learn temporal correspondences.

Text-image pairs present a different but related problem. The caption "a cat sitting on a red sofa" corresponds to the entire image, not to individual patches. But different patches carry different amounts of information relevant to the caption: the cat region matters most, the sofa matters somewhat, and the background barely matters at all. Learning which visual regions to align with which textual concepts is the fine-grained alignment problem, and most large-scale contrastive models only partially solve it by training on whole image-caption pairs.

The alignment problem also manifests in training data quality. Alt-text written by web authors to describe images varies enormously in quality and specificity. Some alt-text says "a dog jumping over a fence in a park," which precisely describes the image. Some says "image_1234.jpg," which is metadata with no visual correspondence. Some says "our dog, Charlie, at his favorite park," which conveys sentiment and personal context but little about the visual content. Models must extract useful alignment signal from this noisy mixture.

Raw signals from different modalities carry semantic information at very different levels of abstraction. A pixel value of 127 in the red channel is a very low-level measurement. The word "crimson" is a high-level semantic concept. To align "red" with pixels that depict red objects, the visual encoder must have already learned to group pixels by color, identify regions, separate objects from backgrounds, and understand that redness is a property of objects rather than pixels in isolation. This requires learning rich visual semantics before any meaningful cross-modal alignment can happen.

The semantic gap is why training multimodal models from scratch on raw pixels and raw text is much harder than composing pretrained unimodal encoders. A large pretrained visual encoder like ResNet or ViT has already learned to extract semantically meaningful features: activations in late layers respond to concepts like "dog," "outdoor scene," or "circular object" rather than raw pixel statistics. Starting multimodal training from these rich features dramatically narrows the semantic gap because the visual representations are already operating at a level of abstraction comparable to language.

This insight explains why the CLIP architecture, which we study in the next chapter, uses large pretrained encoders as its starting point rather than training from scratch. The encoder pretraining is where semantic richness is built; multimodal training then teaches the encoders to align their representations across modalities.

Paired multimodal data is harder to collect than unimodal data. Building a large image-text dataset by scraping alt-text from the web produces pairs with highly variable quality. Many captions describe images poorly, omit key visual content, or convey metadata about the image rather than its visual content. Audio transcripts from automatic speech recognition contain word errors and omit prosodic information. Video subtitles are often temporally misaligned with the speech they transcribe.

This noise fundamentally affects what the model can learn. If 20% of training pairs have captions that are semantically unrelated to the image, training on those pairs produces contradictory gradient signals. The model learns to associate visual features with captions that do not describe them, degrading the quality of the joint embedding space. Filtering and cleaning multimodal training data is therefore a necessary part of the training pipeline for any serious multimodal model.

Multimodal models must tolerate noisy correspondence and learn useful representations even when the pairing between modalities is imperfect. Contrastive training is particularly well-suited to this: when the noise rate is not too high, the correct positive pairs will on average receive stronger gradient signal than the random noise pairs, and the model learns despite the noise. At high noise rates, however, this tolerance breaks down.

The foundational idea in multimodal representation learning is the joint embedding space: a shared high-dimensional vector space where representations from all modalities can coexist, and where semantically similar content is geometrically close regardless of which modality it came from.

Formally, let $f_V : \mathbb{R}^{H \times W \times 3} \to \mathbb{R}^d$ be a visual encoder and $f_T : \mathcal{V}^* \to \mathbb{R}^d$ be a text encoder, both mapping into the same $d$-dimensional space. The goal of joint embedding learning is to train $f_V$ and $f_T$ so that for a matched image-text pair $(I, T)$:

and for a mismatched pair $(I, T')$ where $T'$ describes a different image:

The "approximately equal" and "not approximately equal" here are measured by a distance or similarity function. In practice, the most common choice is cosine similarity:

where:

• $u, v \in \mathbb{R}^d$ are embedding vectors
• $u \cdot v$ is the dot product (sum of element-wise products)
• $\|u\|$ and $\|v\|$ are the Euclidean (L2) norms of the vectors
• The result lies in $[-1, +1]$

Cosine similarity is invariant to the magnitude of the vectors, focusing purely on their direction in embedding space. Two vectors that point in the same direction have cosine similarity 1 regardless of their lengths, and two vectors that point in opposite directions have cosine similarity $-1$.

In practice, it is common to L2-normalize all embedding vectors to unit length before computing cosine similarity. After normalization, cosine similarity reduces to the dot product:

This simplification is computationally convenient and makes the similarity computation fully expressible as a matrix multiplication, which GPUs handle extremely efficiently.

Cosine similarity has properties that make it particularly well-suited for comparing learned embeddings. First, it normalizes for vector magnitude. This means that if one encoder produces embeddings with larger typical norms than the other, the similarity computation is not biased by this scale difference. The encoders are free to learn embeddings of different magnitudes without that affecting the comparison.

Second, cosine similarity is efficient to compute at scale. If you have a batch of $N$ image embeddings arranged as rows of a matrix $U \in \mathbb{R}^{N \times d}$ and a batch of $N$ text embeddings as rows of $V \in \mathbb{R}^{N \times d}$, both L2-normalized, then the full $N \times N$ pairwise similarity matrix is:

This single matrix multiplication gives you every pairwise similarity between every image and every text in the batch simultaneously. On a modern GPU, this takes milliseconds even for batches of thousands. The efficiency of this computation is what makes it practical to train with very large batches, which, as we will see, is necessary for contrastive learning.

Third, cosine similarity corresponds to an angle in high-dimensional space. Training toward high cosine similarity for matched pairs is equivalent to pushing the angle between the two embedding vectors toward zero, causing them to point in the same direction. This angular interpretation is geometrically clean and has a natural connection to directional statistics in high dimensions.

In a well-trained joint embedding space, the geometry encodes semantic relationships in a principled way. Matched image-text pairs cluster together. Semantically related images cluster together even without text. Visual categories form distinct regions of the space that are consistent with textual concept regions. Two images of cats are close to each other and to text captions describing cats, even if the two images look very different and the two captions use different words.

This geometry is what makes joint embeddings powerful for downstream tasks. Once trained, the same embedding space supports many tasks without task-specific fine-tuning:

• Zero-shot classification: Embed the image and compute its similarity to text descriptions of each class. Classify by the closest class description.
• Cross-modal retrieval: Embed a text query and retrieve the images with the highest cosine similarity.
• Semantic search: Embed images and texts into a unified index and search across modalities simultaneously.
• Concept probing: Embed a concept phrase and find the images that most strongly embody it, even without explicit training on that concept.

The joint embedding space thus acts as a universal semantic index, a map where proximity means meaning similarity regardless of modality of origin.

An important and initially surprising property of joint embedding spaces is that even after successful contrastive training, image and text embeddings tend to occupy different regions of the space rather than fully intermixing. This is called the modality gap.

Research has shown that image embeddings form a tight cone in one part of the embedding sphere, and text embeddings form a different cone in another part. Matched pairs are "close" in the sense that their image and text cones are adjacent, but within each cone, the two modalities are not interleaved. You would not mistake an image embedding for a text embedding by examining its position in the space.

The modality gap arises because the two encoders develop different geometric organization internally. The contrastive objective only enforces that matched pairs are closer than mismatched pairs. It does not require that images and texts with identical meanings produce exactly the same vector. As long as the relative ordering is correct, the absolute positions can differ. The gap is a stable geometric structure that emerges consistently across different model sizes and training regimes.

The modality gap has practical consequences. Arithmetic operations that assume images and texts share the same manifold, such as "visual embedding + text embedding" to compose visual and linguistic concepts, may produce vectors that fall outside either cone and carry confused semantics. Techniques like SigLIP and CLIPA propose modifications to the training objective to reduce the gap, but it has not been fully eliminated in any current system.

Learning a joint embedding space requires a training objective that enforces alignment between modalities. The dominant approach in modern multimodal learning is contrastive learning with a contrastive loss applied over batches of paired data.

Contrastive learning operates on matched pairs (positives) and unmatched pairs (negatives). Given a dataset of matched image-text pairs $\{(I_i, T_i)\}_{i=1}^N$, the training objective rewards the model for placing matched pairs close together in embedding space while pushing unmatched pairs apart.

The key insight that makes contrastive learning efficient is the in-batch negatives strategy. Within a single training batch of $N$ pairs, there are $N$ positive pairs: the matched $(I_i, T_i)$ for each $i$. There are also $N(N-1)$ negative pairs: every combination of image $i$ with text $j$ where $i \neq j$. A single batch of $N$ pairs thus generates $O(N^2)$ training signal, making contrastive learning very data-efficient. A batch of 256 pairs gives 256 positives and 65,280 negatives. A batch of 32,768 pairs gives over a billion negatives.

This in-batch negative structure is efficient but introduces a subtle assumption: the negatives are truly unmatched. If the training data contains near-duplicate images or texts, some of the "negative" pairs within a batch may be semantically matched despite being labeled as negatives. This false negative problem degrades training quality and is an active area of research.

The standard loss used in CLIP and most modern multimodal contrastive models is InfoNCE (Noise-Contrastive Estimation), applied symmetrically across both modalities. InfoNCE frames the problem as classification: given an image embedding $v_i$, classify which of the $N$ text embeddings in the batch is the matched one.

The loss for the image-to-text direction treats each image as a query and the $N$ texts as classes:

This is a cross-entropy loss where the correct class is index $i$ (the matched text). The denominator sums over all texts in the batch, making the loss equivalent to classification over $N$ classes.

Symmetrically, the text-to-image loss treats each text as a query:

The full symmetric InfoNCE loss averages both directions:

where:

• $v_i = f_V(I_i)$ is the L2-normalized visual embedding of image $i$
• $t_i = f_T(T_i)$ is the L2-normalized text embedding of text $i$
• $\tau > 0$ is a learnable temperature parameter controlling softmax sharpness
• $N$ is the batch size
• The inner sums run over all $N$ texts or images in the batch

The symmetry is important: the first term trains the visual encoder to produce embeddings that are discriminable from the text side, and the second term trains the text encoder to produce embeddings that are discriminable from the visual side. Neither encoder is treated as the "anchor"; both are jointly optimized.

The temperature parameter $\tau$ controls the sharpness of the softmax distribution over negatives, and its effect on training is substantial.

When $\tau$ is large (close to 1 or above), the softmax distribution is nearly uniform: all pairs receive similar probability mass and the loss treats positives and negatives almost equally. The gradients are small and training is slow.

When $\tau$ is small (close to 0), the softmax distribution is very peaked: the pair with the highest similarity receives nearly all probability mass. This sharpens the learning signal but can also cause instability if the embeddings are poorly organized early in training, since the softmax saturates quickly and produces vanishing gradients.

In practice, $\tau$ is learned jointly with the encoders. CLIP initializes $\tau$ as a learnable scalar with initial value $\exp(\log(0.07)) \approx 0.07$ and clips it at $\exp(\log(100)) = 100$ to prevent instability. The model learns to set $\tau$ to a value that produces useful gradient signal given the current quality of the embeddings.

Intuitively, as training progresses and the embeddings improve, positive pairs become much more similar than negative pairs, and a smaller $\tau$ is needed to maintain strong learning signal. Early in training when embeddings are near-random, a larger $\tau$ prevents the loss from exploding by keeping the distribution less peaked. The ability to learn $\tau$ allows the training to self-regulate this tradeoff automatically.

InfoNCE has a principled probabilistic interpretation: maximizing the InfoNCE objective is equivalent to maximizing a lower bound on the mutual information between the visual and textual representations. Specifically, if we denote the visual and textual representations as random variables $V$ and $T$, then:

where $I(V; T)$ is the mutual information between the representations and $\mathcal{L}_{\text{InfoNCE}}$ is the InfoNCE loss value. Minimizing the InfoNCE loss therefore drives the representations to share as much information as possible, creating the mutual information bottleneck that enforces semantic alignment. This connection motivates why contrastive objectives produce semantically rich representations: they are explicitly optimized to capture shared information between modalities.

A key challenge in contrastive training is that random negatives are often too easy to distinguish. If you pair a picture of a sunset with a caption about "a factory in winter," the model can trivially distinguish positive from negative based on superficial features without learning any fine-grained semantics. Easy negatives provide weak gradient signal because the loss is already near-minimal even with poor embeddings.

Hard negatives are pairs that are semantically close but factually mismatched. A picture of a golden retriever paired with a caption about a labrador retriever is a hard negative: both involve dogs, both have similar visual and textual features, but they are different pairs. Learning to distinguish hard negatives pushes the model to encode fine-grained semantic detail rather than coarse categorical features.

Several approaches address hard negative mining:

• Large batch sizes: Larger batches contain more total pairs, increasing the probability that some off-diagonal pairs are semantically similar to the positive pair. CLIP uses batches of up to 32,768 pairs precisely because this greatly increases the density of hard negatives.
• Explicit hard negative sampling: Mining specifically difficult negative pairs based on current embedding similarity before each training step. Pairs where the model assigns high similarity but no ground-truth match constitute informative negatives.
• Data augmentation: Creating pairs that share partial content, forcing the model to attend to specific discriminative details to correctly identify the match.
• Curriculum learning: Starting with easy negatives and gradually increasing difficulty as training progresses, mirroring how humans learn by scaffolding difficulty.

The tradeoff with hard negatives is that if negatives are too hard, false negatives (semantically matched pairs that are labeled as negatives) become more likely. At very large batch sizes, some "negative" image-text pairs in the batch may describe the same concept. These false negatives degrade training by penalizing correct associations.

Given modality-specific encoders and the alignment objective, there are several architectural patterns for building multimodal models. These differ in when and how the modalities interact and in what tradeoffs they make between efficiency and representational power.

The simplest architecture processes each modality independently with separate encoders and combines them only at the final embedding level via a similarity computation. This is the architecture used by CLIP.

The visual encoder processes the image through a series of transformer or convolutional layers and produces a single image embedding vector. The text encoder processes the text through a series of transformer layers and produces a single text embedding vector. The two embeddings are compared using cosine similarity. The encoders never attend to each other; they interact only through the scalar similarity score at the very end.

The advantage of this architecture is computational efficiency. Because the encoders never attend to each other, embeddings can be computed independently for any image and any text. This means you can precompute and cache embeddings for an entire database of images or documents offline. At query time, you compute only the query embedding and perform a single nearest-neighbor lookup in the precomputed database. For a database of one million images, this takes milliseconds rather than the minutes it would take if you had to run the full model over every image at query time.

The disadvantage of the dual-encoder is limited cross-modal reasoning. When each modality is processed in isolation, the model cannot use textual context to guide visual attention, or visual context to disambiguate textual meaning. The only interaction is the global similarity score, which is a single number summarizing the relationship between two entire inputs. This global summary is insufficient for tasks that require fine-grained correspondence: the model cannot answer "is the person in this image the one mentioned in this text?" because it only knows the overall similarity, not which visual regions correspond to which textual descriptions.

Cross-encoders process both modalities jointly using cross-attention. In the most common formulation, the image is divided into patches using a ViT-style tokenization, creating a sequence of visual tokens. These visual tokens are concatenated with the text tokens, and a transformer processes the combined sequence with full self-attention across both modalities simultaneously.

This enables rich interaction: text tokens can attend to image patches and image patches can attend to text tokens. A text token representing "cat" can attend strongly to the image patch depicting the cat and weakly to background patches. An image patch depicting a face can attend strongly to any text token describing facial features. This fine-grained cross-modal attention produces much richer representations than the dual-encoder and typically outperforms it on tasks requiring detailed visual understanding or grounding.

The cost of cross-encoders is computational. The representation of each modality depends on the other, so embeddings cannot be precomputed independently. For retrieval over large databases, this means running the cross-encoder over every candidate image-text pair at query time, which is prohibitively expensive at scale. A database of one million images would require one million cross-encoder forward passes per query, taking minutes rather than milliseconds.

Many modern systems therefore use a two-stage pipeline: a dual-encoder for fast coarse retrieval that narrows a database of millions to a shortlist of tens or hundreds of candidates, followed by a cross-encoder reranker that performs detailed reasoning over the shortlist. This combines the scalability of dual-encoders with the accuracy of cross-encoders.

Between the extremes of late fusion (dual-encoder, modalities interact only at the similarity computation) and early fusion (cross-encoder, modalities interact at every layer), several intermediate architectures offer different tradeoffs.

Mid-fusion architectures process each modality separately for several early layers, then concatenate the intermediate representations and process them jointly through later layers. The early layers learn modality-specific features, and the later shared layers learn cross-modal interactions. This balances the speed benefits of independent processing with the expressive power of cross-modal attention.

Perceiver-style cross-attention introduces a fixed set of latent query vectors that attend to features from both modalities via cross-attention. The latent vectors serve as a bottleneck that forces the model to compress cross-modal information into a compact representation. The number of latent vectors is fixed and independent of the input lengths, making the computational cost predictable even for very long inputs. This architecture was popularized by the Perceiver and Flamingo models.

Prefix-based fusion prepends visual embeddings as prefix tokens to a language model, letting the language model to generate text conditioned on visual content. The visual encoder produces a sequence of patch embeddings, a linear projection maps these into the language model's token embedding space, and these projected embeddings are prepended to the text token sequence. The language model can then attend to visual tokens using its standard self-attention. This is the approach used by LLaVA and InstructBLIP, which we examine in later chapters of this part.

The most powerful form of cross-modal interaction is cross-modal attention, a direct extension of the self-attention mechanism you studied in the attention chapters of this handbook. Understanding it precisely will help you read papers on multimodal transformers.

Recall from our earlier discussion of self-attention that it computes a weighted combination of value vectors, where the weights are determined by the similarity between query vectors and key vectors:

In self-attention, all three matrices $Q$, $K$, $V$ are derived from the same input sequence by linear projections. For a sequence $X \in \mathbb{R}^{N \times d_\text{model}}$:

Each token queries and answers questions about other tokens in the same sequence. This is appropriate for processing a single modality but does not allow tokens from one modality to attend to tokens from another.

Cross-modal attention modifies this by letting queries come from one modality and keys and values come from the other. Specifically, to attend from text to visual features, we use:

• Queries $Q$ from the text representation: $Q = X_T W_Q$ where $X_T$ is the text token sequence
• Keys $K$ and Values $V$ from the visual representation: $K = X_V W_K$ and $V = X_V W_V$ where $X_V$ is the visual token sequence

The attention output is then:

where:

• $Q \in \mathbb{R}^{N_T \times d_k}$ is the query matrix from $N_T$ text tokens with query dimension $d_k$
• $K \in \mathbb{R}^{N_V \times d_k}$ is the key matrix from $N_V$ visual tokens with key dimension $d_k$
• $V \in \mathbb{R}^{N_V \times d_v}$ is the value matrix from $N_V$ visual tokens with value dimension $d_v$
• The output has shape $\mathbb{R}^{N_T \times d_v}$, one attended feature per text token

Each text token attends to all visual tokens, creating a weighted combination of visual values that is most relevant to that text token's query. A text token representing "cat" will attend strongly to image patches containing cat-like features and weakly to patches containing irrelevant background content. This soft, differentiable attention mechanism is precisely how cross-modal alignment is computed through the transformer architecture.

The scaling factor $\frac{1}{\sqrt{d_k}}$ prevents the dot products from growing large as $d_k$ increases, which would otherwise cause softmax to saturate in a region of very small gradients. Recall from the attention chapters that this scaling is necessary for stable training, especially with larger key dimensions.

Many architectures apply cross-attention in both directions: text-to-image and image-to-text. This allows each modality to gather information from the other, creating representations where every token is cross-modally informed.

For text-to-image attention, queries come from the text representation and keys and values come from the image representation. Each text token learns which image regions are most relevant to its meaning. The output is a set of image-conditioned text representations where each text token has absorbed relevant visual context.

For image-to-text attention, queries come from the image representation and keys and values come from the text representation. Each image patch learns which text tokens are most relevant to its visual content. The output is a set of text-conditioned image representations where each patch has absorbed relevant linguistic context.

When combined with self-attention within each modality, this bidirectional cross-attention produces richly cross-informed representations. Each token has access to relevant content from both its own modality, through self-attention, and from the other modality, through cross-attention. The interplay between self-attention and cross-attention is what gives cross-encoder architectures their expressive power.

Let us walk through the computation of the InfoNCE loss for a small batch of three image-text pairs to make the mathematics concrete. This example contains every step with explicit numbers so that the relationship between the geometry of the embedding space and the value of the loss is clear.

Suppose after encoding and L2-normalizing, we have embeddings in $\mathbb{R}^2$ (two dimensions for visualization purposes):

• Image embeddings: $v_1 = [1, 0]$, $v_2 = [0, 1]$, $v_3 = [-1, 0]$ (all unit vectors)
• Text embeddings: $t_1 = [0.9, 0.4]$ (similar to $v_1$), $t_2 = [-0.3, 0.95]$ (similar to $v_2$), $t_3 = [-0.9, -0.4]$ (similar to $v_3$), all after L2-normalization.

where $(v_i, t_i)$ are matched pairs.

Step 1: Compute the similarity matrix $S = V T^T$.

For our example (using approximate values after normalizing $t_1, t_2, t_3$):

The diagonal entries are large and positive (high similarity for matched pairs) while off-diagonal entries are smaller or negative (low similarity for mismatched pairs).

Step 2: Scale by temperature $\tau = 0.1$.

The temperature scaling amplifies the differences, making the softmax much more peaked.

Step 3: Compute row-wise softmax for the image-to-text direction.

For image $i = 1$:

Since $\exp(9.0) \approx 8103$ and $\exp(-3.0) \approx 0.05$ and $\exp(-9.0) \approx 0.0001$, the softmax is approximately $[0.9999, 0.0001, 0]$. Almost all probability mass sits on the matched pair (index 1).

Step 4: Compute cross-entropy loss for image 1.

Very low loss because the matched pair has very high similarity and temperature scaling made the difference extremely sharp.

Step 5: Repeat symmetrically for text-to-image. The structure is identical by the construction of our example.

Step 6: Average all terms to get the final loss. In this idealized example, after several steps of training that pushed embeddings toward high similarity for matched pairs, the loss is very near zero.

Now contrast this with an untrained model where embeddings are random. Random unit vectors in high dimensions have expected cosine similarity near zero, so all entries of $S$ are near zero and the softmax distribution is nearly uniform. In that case, $p_{ii} \approx 1/N$ and the loss is approximately $\log N$, which for a batch of 32,768 is about 10.4. Training drives the loss from $\log N$ toward zero.

Let us implement a minimal multimodal contrastive learning system from scratch to build concrete intuition for how joint embedding spaces are trained.

We start by creating synthetic image and text embeddings to simulate what pretrained encoders would produce.

The synthetic data mimics real multimodal data: image and text features live in different subspaces (the rotation matrix simulates the modality gap) but share both semantic-category structure and a weaker pair-specific signal. The rotation ensures the raw features are not directly comparable, requiring the model to learn to align them.

We implement a simple dual-encoder with separate projection heads for each modality.

The ModalityProjection module acts as a bridge between each modality's raw feature space and the shared joint embedding space. The two-layer MLP with a ReLU nonlinearity gives the projections enough capacity to learn nonlinear transformations. The L2-normalization at the end ensures all output embeddings lie on the unit hypersphere, which is the standard for contrastive learning.

The loss decreases substantially over training, showing the encoders have learned to map matched image-text pairs to similar regions of the joint embedding space. The learned temperature reflects how sharply the model discriminates between positives and negatives: a temperature near 0.1 indicates the model has learned to produce high similarity for matched pairs and near-zero or negative similarity for mismatched pairs.

A direct measure of alignment quality is retrieval accuracy: given each image embedding, can we retrieve its matched text from the full pool of all texts?

Recall at $k$ (R@$k$) measures the fraction of queries for which the correct match appears in the top-$k$ retrieved results. High R@1 indicates tight alignment: the correct match is the single most similar item. High R@5 or R@10 indicates the model is directionally correct but may not achieve perfect precision at rank 1. In practice on large-scale datasets like MS-COCO, state-of-the-art models achieve R@1 image-to-text retrieval above 80%.

The most intuitive way to understand what the joint embedding space has learned is to project it to two dimensions and examine the cluster structure.

Circles represent image embeddings and triangles represent text embeddings. Points of the same color belong to the same semantic category. When contrastive training succeeds, circles and triangles of the same color cluster near each other, showing that the image and text encoders have learned to map semantically equivalent content to the same region of the joint space. The five-category structure should be visible as five distinct clusters, with image and text embeddings from each category interleaved rather than separated.

The diagonal of this heatmap represents matched pairs: each image compared with its corresponding text. The amber outlines make those exact pairs explicit. Because the synthetic dataset also has five shared semantic categories, some off-diagonal cells are dark as well: these are unmatched examples that remain semantically related. The model therefore learns both exact-pair alignment and broader category neighborhoods, which explains why retrieval rank remains a stricter test than category separation alone.

A key diagnostic for contrastive learning is the distribution of cosine similarities for positive (matched) and negative (unmatched) pairs. Good alignment produces a clear separation between these two distributions.

Before training, positive and negative pair similarities are indistinguishable: the encoders have not yet learned what makes an image-text pair match. After training, positive pairs cluster near 1.0 and most negatives remain much lower. The smaller negative peak near 1.0 contains semantically related examples from the same synthetic category. This is a useful geometric signature: the space captures broad meaning while the exact ordering within a semantic neighborhood determines downstream retrieval quality.

The code example above uses small synthetic data to illustrate the mechanism. Real multimodal pretraining operates at very different scales, and the scaling decisions strongly affect model quality.

CLIP was trained on 400 million image-text pairs assembled from the web. ALIGN used 1.8 billion pairs. These massive datasets are necessary because the model must learn a joint embedding space that covers the enormous diversity of visual and linguistic concepts in the world. With small datasets, the model only learns to align the specific visual and textual patterns present in training and fails to generalize to novel content.

Dataset diversity is as important as scale. A dataset of 400 million similar images, such as product photography from a single retailer, would produce much weaker representations than 400 million diverse images spanning art, nature, sports, scientific imagery, and everyday scenes. Diversity in training data leads to generalization across the broad semantic space the model must cover.

The tradeoff between scale and quality is not settled. LAION-5B collected five billion image-text pairs through aggressive web scraping with minimal filtering, while DataComp curated datasets of varying sizes with different quality thresholds. Research on DataComp showed that smaller, carefully curated datasets can match or exceed the quality of much larger noisily collected datasets on downstream benchmarks. The lesson: scale matters, but quality at scale matters more.

Contrastive learning particularly benefits from large batch sizes because larger batches contain more negative pairs. With a batch of 32,768 pairs, each image competes against 32,767 other texts during training, forcing the model to learn very precise discrimination. A batch of 64 pairs, by contrast, provides only 63 negatives per query, which is usually insufficient for learning fine-grained semantics.

CLIP's batch size of 32,768 was enabled by distributing training across hundreds of GPUs and aggregating embeddings across all devices before computing the loss. In distributed contrastive training, each device computes embeddings for its local batch shard, then all-gathers the embeddings across all devices to construct the global similarity matrix. This global-gather pattern makes the effective batch size for the loss computation equal to the total batch size across all devices rather than the local shard size.

The computational cost of this global-gather scales linearly with the number of devices, but the resulting large batch dramatically improves learning efficiency. The CLIP training recipe is fundamentally bottlenecked by the batch size: more pairs means more negatives, means sharper discrimination, means better alignment.

CLIP experiments show that visual encoders ranging from ResNet-50 (25 million parameters) to ViT-L/14 (307 million parameters) consistently benefit from scale, with larger encoders creating higher-quality embeddings on zero-shot classification and retrieval benchmarks. Text encoders of comparable scale are used alongside the visual encoders.

The choice of visual encoder architecture also matters. ResNets apply convolutional filters in a translationally equivariant way, which is well-suited to natural image statistics where objects can appear at any spatial position. ViTs divide the image into patches and process them with self-attention, which is more flexible but requires more data to learn good representations from scratch. At sufficient scale, ViTs typically outperform ResNets, which is why the largest CLIP models use ViT encoders.

Multimodal learning has produced remarkable capabilities, but it comes with basic limitations that researchers are actively working to address.

Even after contrastive training, image and text embeddings tend to occupy different regions of the joint embedding space rather than fully intermixing. This is the modality gap. Research shows that image embeddings cluster in a tight cone in one part of the embedding hypersphere, and text embeddings cluster in a different cone. Matched pairs have their cones adjacent, but within each cone the two modalities do not interleave. The gap arises because the two encoders develop different geometric organization internally, and the contrastive objective only requires relative ordering (matched pairs closer than mismatched pairs) rather than exact positional overlap.

The modality gap has practical consequences for applications that assume images and texts share the same geometric manifold. Composing visual and textual concepts via vector arithmetic may produce vectors that fall outside either cone. Techniques like SigLIP propose modifications to the contrastive loss that reduce the gap by encouraging images and texts to occupy the same region of space, but the gap has not been fully eliminated in current systems.

Contrastive models trained on whole-image, whole-caption pairs learn global correspondences but struggle with fine-grained compositional understanding. The captions "a red ball to the left of a blue cube" and "a blue ball to the left of a red cube" may receive very similar scores from a CLIP model because the model attends to the presence of red, blue, ball, and cube without correctly reasoning about the spatial relations between them.

This limitation is well-documented by benchmarks like Winoground and ARO (Attribution, Relation, Order), which test compositional understanding by asking models to distinguish between captions that share the same words but differ in their grammatical structure and meaning. Current contrastive models perform surprisingly poorly on these benchmarks, sometimes approaching chance level, even while achieving excellent performance on standard image classification and retrieval.

The failure stems from how the training objective and the global embedding approach interact. Because the loss only cares about whether the correct image-text pair is ranked highest, the model can learn to match correct pairs without learning the compositional structure that distinguishes them from similar-but-wrong pairs. Dense alignment methods, which learn patch-level alignments between visual regions and textual phrases rather than global image-caption correspondences, partially address this by requiring the model to establish finer-grained correspondences.

Web-scraped image-text pairs reflect the biases of the web: overrepresentation of certain demographics, cultural perspectives, and domains, and underrepresentation of others. Models trained on such data inherit these biases and can exhibit harmful associations. CLIP has been shown to exhibit association biases which reflects social stereotypes present in web images and text, including associations between demographic groups and occupational roles that do not reflect actual proportions.

Curating higher-quality data and debiasing training data are active areas of research, but there is no complete solution. The scale required for good multimodal models, hundreds of millions to billions of pairs, makes manual curation impractical. Automated filtering can remove the most obvious low-quality pairs, but subtle biases are much harder to detect and remove at scale. Users of pretrained multimodal models should be aware of these limitations when deploying in sensitive applications and evaluate for bias on their specific use cases.

Standard contrastive models produce a single embedding for an entire image. They cannot answer questions about specific objects or regions, localize referred objects, or perform fine-grained visual question answering. Extending multimodal models to support grounding, the ability to connect textual references to specific image regions, requires architectural modifications beyond the dual-encoder.

Grounding requires the model to produce region-level or patch-level representations that can be associated with specific noun phrases or entity mentions. Models like OFA, Grounding DINO, and region-level contrastive approaches address this by training with region-text pairs or bounding box supervision alongside image-level pairs. These additional supervision signals force the model to learn spatial correspondences that pure global contrastive training cannot provide.

Extending joint embeddings to video introduces challenges that static image-text models cannot handle. A video is a temporal sequence of frames, and many captions describe temporal events: "a person picks up a ball and throws it." Correct alignment requires the model to understand the ordering and causality of events rather than merely detecting the presence of visual concepts.

Static contrastive models trained on video frames or clips can match global descriptions to global clips, but they struggle with temporal ordering and causal relationships. Video-specific architectures like VideoCLIP and TimeSformer introduce temporal attention that processes frames in sequence rather than treating them as a bag of independent images. Extending this to event-level temporal alignment, where specific parts of a video correspond to specific parts of a description, remains an open research problem.

This chapter laid the groundwork for understanding multimodal AI systems, from the nature of modalities through the mathematics of joint embeddings to the architectural patterns and limitations of current systems. The key ideas to carry forward are:

• Modalities are distinct channels of information (images, text, audio, video) with their own structure and native representations. Each modality requires modality-specific encoders before cross-modal alignment can occur.
• The core challenges of multimodal learning are the representation gap (different native spaces), the alignment problem (correspondence between modalities is fuzzy), the semantic gap (raw signals carry meaning at different levels of abstraction), and data heterogeneity and noise.
• Joint embedding spaces are shared $d$-dimensional vector spaces where semantically similar content from different modalities is placed close together. Cosine similarity measures closeness, and the joint space enables zero-shot retrieval, classification, and cross-modal search without task-specific fine-tuning.
• The modality gap is an emergent property of joint embedding spaces where image and text embeddings occupy different regions of the hypersphere even after successful training. It arises because the contrastive objective enforces relative ordering rather than exact positional overlap.
• Contrastive learning with the symmetric InfoNCE loss is the dominant training objective. It rewards matched pairs for high similarity and penalizes mismatched pairs, using all in-batch pairs as efficient negatives and generating $O(N^2)$ training signal from $N$ pairs. The temperature parameter controls softmax sharpness and is learned jointly with the encoders.
• Dual-encoder architectures process modalities independently and are highly efficient for large-scale retrieval through precomputed embedding indices. Cross-encoders process modalities jointly with attention and enable richer reasoning but cannot precompute embeddings independently, limiting their scalability.
• Cross-modal attention extends self-attention by using queries from one modality and keys and values from the other, letting each token to gather information from the other modality. Bidirectional cross-modal attention, combined with within-modality self-attention, produces fully cross-informed representations.
• Hard negatives are necessary for learning fine-grained semantics. Large batch sizes increase hard negative density naturally; explicit hard negative mining and curriculum learning can improve efficiency further.
• Limitations include the modality gap, difficulty with fine-grained compositional reasoning, inherited data bias, weak spatial grounding, and challenges with temporal reasoning in video. These are active areas of research.

With these foundations in place, you are ready to understand CLIP in the next chapter, which instantiates these ideas at scale with architectural and training decisions that became the blueprint for the field. The chapters that follow examine visual instruction tuning and full vision-language architectures that build on the joint embedding foundation. These models support open-ended generation and visual grounding within multimodal dialogue.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about multimodal foundations and cross-modal alignment.