Multimodal Foundations › Joint Embedding Spaces

The foundational idea in multimodal representation learning is the joint embedding space: a shared high-dimensional vector space where representations from all modalities can coexist, and where semantically similar content is geometrically close regardless of which modality it came from.

Formally, let $f_V : \mathbb{R}^{H \times W \times 3} \to \mathbb{R}^d$ be a visual encoder and $f_T : \mathcal{V}^* \to \mathbb{R}^d$ be a text encoder, both mapping into the same $d$-dimensional space. The goal of joint embedding learning is to train $f_V$ and $f_T$ so that for a matched image-text pair $(I, T)$:

$$f_V(I) \approx f_T(T)$$

and for a mismatched pair $(I, T')$ where $T'$ describes a different image:

$$f_V(I) \not\approx f_T(T')$$

The "approximately equal" and "not approximately equal" here are measured by a distance or similarity function. In practice, the most common choice is cosine similarity:

$$\text{sim}(u, v) = \frac{u \cdot v}{\|u\| \|v\|}$$

where:

• $u, v \in \mathbb{R}^d$ are embedding vectors
• $u \cdot v$ is the dot product (sum of element-wise products)
• $\|u\|$ and $\|v\|$ are the Euclidean (L2) norms of the vectors
• The result lies in $[-1, +1]$

Cosine similarity is invariant to the magnitude of the vectors, focusing purely on their direction in embedding space. Two vectors that point in the same direction have cosine similarity 1 regardless of their lengths, and two vectors that point in opposite directions have cosine similarity $-1$.

In practice, it is common to L2-normalize all embedding vectors to unit length before computing cosine similarity. After normalization, cosine similarity reduces to the dot product:

$$\text{sim}(u, v) = \hat{u} \cdot \hat{v} \quad \text{where} \quad \hat{u} = \frac{u}{\|u\|}, \;\; \hat{v} = \frac{v}{\|v\|}$$

This simplification is computationally convenient and makes the similarity computation fully expressible as a matrix multiplication, which GPUs handle extremely efficiently.

Cosine similarity has properties that make it particularly well-suited for comparing learned embeddings. First, it normalizes for vector magnitude. This means that if one encoder produces embeddings with larger typical norms than the other, the similarity computation is not biased by this scale difference. The encoders are free to learn embeddings of different magnitudes without that affecting the comparison.

Second, cosine similarity is efficient to compute at scale. If you have a batch of $N$ image embeddings arranged as rows of a matrix $U \in \mathbb{R}^{N \times d}$ and a batch of $N$ text embeddings as rows of $V \in \mathbb{R}^{N \times d}$, both L2-normalized, then the full $N \times N$ pairwise similarity matrix is:

$$S = U V^T$$

This single matrix multiplication gives you every pairwise similarity between every image and every text in the batch simultaneously. On a modern GPU, this takes milliseconds even for batches of thousands. The efficiency of this computation is what makes it practical to train with very large batches, which, as we will see, is necessary for contrastive learning.

Third, cosine similarity corresponds to an angle in high-dimensional space. Training toward high cosine similarity for matched pairs is equivalent to pushing the angle between the two embedding vectors toward zero, causing them to point in the same direction. This angular interpretation is geometrically clean and has a natural connection to directional statistics in high dimensions.

In a well-trained joint embedding space, the geometry encodes semantic relationships in a principled way. Matched image-text pairs cluster together. Semantically related images cluster together even without text. Visual categories form distinct regions of the space that are consistent with textual concept regions. Two images of cats are close to each other and to text captions describing cats, even if the two images look very different and the two captions use different words.

This geometry is what makes joint embeddings powerful for downstream tasks. Once trained, the same embedding space supports many tasks without task-specific fine-tuning:

• Zero-shot classification: Embed the image and compute its similarity to text descriptions of each class. Classify by the closest class description.
• Cross-modal retrieval: Embed a text query and retrieve the images with the highest cosine similarity.
• Semantic search: Embed images and texts into a unified index and search across modalities simultaneously.
• Concept probing: Embed a concept phrase and find the images that most strongly embody it, even without explicit training on that concept.

The joint embedding space thus acts as a universal semantic index, a map where proximity means meaning similarity regardless of modality of origin.

An important and initially surprising property of joint embedding spaces is that even after successful contrastive training, image and text embeddings tend to occupy different regions of the space rather than fully intermixing. This is called the modality gap.

Research has shown that image embeddings form a tight cone in one part of the embedding sphere, and text embeddings form a different cone in another part. Matched pairs are "close" in the sense that their image and text cones are adjacent, but within each cone, the two modalities are not interleaved. You would not mistake an image embedding for a text embedding by examining its position in the space.

The modality gap arises because the two encoders develop different geometric organization internally. The contrastive objective only enforces that matched pairs are closer than mismatched pairs. It does not require that images and texts with identical meanings produce exactly the same vector. As long as the relative ordering is correct, the absolute positions can differ. The gap is a stable geometric structure that emerges consistently across different model sizes and training regimes.

The modality gap has practical consequences. Arithmetic operations that assume images and texts share the same manifold, such as "visual embedding + text embedding" to compose visual and linguistic concepts, may produce vectors that fall outside either cone and carry confused semantics. Techniques like SigLIP and CLIPA propose modifications to the training objective to reduce the gap, but it has not been fully eliminated in any current system.