Vision-Language Projection › The Projection Problem
Language AI HandbookAll chapters Vision-Language Projection › The Projection ProblemOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To understand why projection is necessary, consider the output of a typical vision encoder. As we discussed in the Vision Transformer chapter, a ViT processes an image by dividing it into patches, commonly 16×16 pixels, encoding each patch into a vector. For a 224×224 image, this yields patch tokens, plus a class token, totaling 197 vectors. If the ViT uses a hidden dimension of 768 (the base configuration), the output is a tensor of shape .
These numbers represent more than abstract dimensions. Each of the 196 patch embeddings captures visual features for a specific 16×16 pixel region of the image, encoding information about edges and textures alongside colors and local patterns in that spatial location. The class token, typically positioned at the beginning of the sequence, aggregates global information about the entire image through the self-attention mechanism during vision encoding. Together, these vectors form a spatial map of visual understanding, where the geometric relationships between tokens correspond to physical proximity in the original image. This spatial structure is fundamentally different from the sequential, syntactic structure of language.
Now consider a modern language model like LLaMA or GPT. These models typically expect input embeddings of much higher dimensionality, often 2048, 4096, or 8192 dimensions. They also process sequences composed of discrete token embeddings drawn from their input vocabulary, not patch embeddings from spatial image regions. These linguistic embeddings occupy a semantic space where vectors represent word meanings, grammatical functions, and contextual relationships between concepts. The geometry of this space reflects linguistic regularities: words with similar meanings cluster together, and vector arithmetic can capture semantic relationships.
The mismatch is not just dimensional. Consider what a language model has internalized about its own embedding space. It has learned, through billions of gradient steps on text prediction, that certain directions in its embedding space correspond to grammatical agreement, others to sentiment polarity, others to named entity type, and so on. When an attention head in a language model processes a sequence of tokens, it interprets vectors within this well-structured space, using learned patterns to decide which tokens are relevant to which others. Feeding the language model a raw ViT patch embedding, without any transformation, would be like inserting a sentence in an unknown language into the middle of an English passage: the model would try to interpret the foreign tokens using English grammar rules, creating gibberish.
The projection problem involves two mismatches:
- Dimension mismatch: Vision embeddings (e.g., 768-dim) differ from LLM input dimensions (e.g., 4096-dim)
- Semantic structure mismatch: Vision outputs represent spatial patches; LLM inputs represent linguistic concepts built through text-only training
The projection layer must resolve both issues simultaneously: mapping from the vision embedding dimension to the language model dimension , and transforming spatial-visual representations into semantic-linguistic representations that the LLM can interpret within its pre-trained context. This transformation is a technical requirement and a semantic necessity. The LLM has learned to reason about vectors in its input space, associating specific regions of that high-dimensional manifold with concepts. The projection layer must ensure that visual signals corresponding to these concepts land in the appropriate regions of the LLM's semantic space, allowing the model to use its linguistic knowledge for visual understanding.
A useful way to think about this is through the lens of coordinate systems. Imagine two experts, a cartographer and a navigator, each with a map of the same territory but using different coordinate systems. The cartographer uses latitude and longitude; the navigator uses bearing and distance. Both maps accurately represent the same geography, but without a conversion function, the navigator cannot use the cartographer's map. The projection layer is precisely this conversion function, a learned transformation that converts from the coordinate system of visual features to the coordinate system of linguistic representations. The underlying territory, the semantic concepts being represented, is the same in both spaces; only the coordinate conventions differ.
Why Representation Spaces Align at All
It is natural to wonder why projection should work at all. If vision encoders and language models are trained on completely different data with different objectives, why should their internal representations have any geometric relationship that a learned projection could exploit?
The answer lies in the shared semantic structure of the world. Both CLIP-style vision encoders and large language models are trained, directly or indirectly, on the internet, which is full of images with captions, articles with photographs, and visual media with textual context. Even if the same pixel data and the same text are not processed by both models simultaneously, the statistical regularities of the world ensure that both spaces develop clusters and directions corresponding to the same semantic categories. A dog looks like a dog whether you describe it in words or depict it in pixels.
More formally, consider a contrastive vision encoder like CLIP, trained to bring image embeddings close to their paired text embeddings. CLIP's vision encoder is explicitly trained to align with a text encoder, meaning its visual representations are already partly linguistic in character. When you use CLIP visual features as input to a projection layer targeting a separate LLM, you are building on partial alignment: CLIP's vision encoder has already been encouraged to organize its representations in a language-compatible way, making the projection's job substantially easier.
For vision encoders not trained contrastively, like those trained purely on image classification, the alignment is less direct but still exploitable. Classification training forces the encoder to represent visual categories, and categories correspond to words. The ImageNet classes, for instance, are linguistic labels, and training a vision encoder to classify into those labels creates representations that correlate with the language model's representations for those same category names. The projection layer, trained on enough image-text pairs, can learn to exploit these correlations even when the original encoders were trained independently.