Vision-Language Projection: Bridging Vision Encoders, LLMs

A vision-language model faces a basic architectural challenge: the vision encoder speaks one language, and the large language model speaks another. When you feed an image through a Vision Transformer, you receive a sequence of patch embeddings, perhaps 197 vectors of 768 dimensions each representing spatial regions of the image. But your language model expects linguistic tokens, perhaps vectors of 4096 dimensions drawn from a vocabulary of subword units. These two representations occupy different vector spaces with different semantic structures. They are, in a sense, orthogonal dialects of meaning. The projection layer is the translator between these modalities. It turns visual representations into the semantic space that the language model understands. This enables the LLM to "read" images as if they were text tokens. Without this bridge, even the most powerful vision encoder and the most capable language model would remain unable to communicate, like two brilliant scholars who speak different languages and lack an interpreter.

This challenge is necessary as multimodal AI systems have moved from research projects to practical tools. Early approaches to vision-language modeling often trained both modalities from scratch or relied on shallow fusion mechanisms that concatenated features without true semantic alignment. However, the modern paradigm uses pre-trained models: vision encoders trained on massive image datasets through self-supervised or contrastive learning, and language models trained on internet-scale text corpora. Each possesses deep, specialized knowledge, but their representations evolved independently without any guarantee of compatibility. The projection layer must therefore harmonize these distinct representation spaces. This keeps visual concepts like "texture," "spatial relationship," or "object part" map appropriately to linguistic concepts like "adjective," "preposition," or "noun phrase."

It is worth pausing to appreciate how remarkable this task is. A vision encoder trained entirely on pixel data has never seen a single word. A language model trained on web pages and books has never processed a pixel. Yet both have converged on internal representations that capture something true about the world. The vision encoder that clusters dogs near other dogs in its embedding space, and the language model that clusters "dog" near "canine" and "puppy" in its own space, have both learned the concept of a dog through radically different data pipelines. The projection layer succeeds precisely because this convergent learning creates a geometric correspondance between the two spaces, a correspondence that can be discovered and formalized through supervised training on image-text pairs.

This chapter explores the mechanisms that create this bridge. We will examine the simplest approach, linear projection, where a single matrix multiplication aligns the dimensionalities. We will then deepen this with multi-layer perceptron projections that introduce non-linear transformations for more complex mappings. We will analyze the Q-Former architecture from BLIP-2, which uses a transformer-based query mechanism to compress variable-length visual features into a fixed set of learnable tokens, addressing the challenge of efficiently feeding visual information to frozen LLMs. We will also discuss the training strategies that make these projections learn effectively, including freezing choices, stage-wise training, and loss masking decisions that determine how gradients flow through the system.

To understand why projection is necessary, consider the output of a typical vision encoder. As we discussed in the Vision Transformer chapter, a ViT processes an image by dividing it into patches, commonly 16×16 pixels, encoding each patch into a vector. For a 224×224 image, this yields $(224/16)^2 = 196$ patch tokens, plus a class token, totaling 197 vectors. If the ViT uses a hidden dimension of 768 (the base configuration), the output is a tensor of shape $[197, 768]$.

These numbers represent more than abstract dimensions. Each of the 196 patch embeddings captures visual features for a specific 16×16 pixel region of the image, encoding information about edges and textures alongside colors and local patterns in that spatial location. The class token, typically positioned at the beginning of the sequence, aggregates global information about the entire image through the self-attention mechanism during vision encoding. Together, these vectors form a spatial map of visual understanding, where the geometric relationships between tokens correspond to physical proximity in the original image. This spatial structure is fundamentally different from the sequential, syntactic structure of language.

Now consider a modern language model like LLaMA or GPT. These models typically expect input embeddings of much higher dimensionality, often 2048, 4096, or 8192 dimensions. They also process sequences composed of discrete token embeddings drawn from their input vocabulary, not patch embeddings from spatial image regions. These linguistic embeddings occupy a semantic space where vectors represent word meanings, grammatical functions, and contextual relationships between concepts. The geometry of this space reflects linguistic regularities: words with similar meanings cluster together, and vector arithmetic can capture semantic relationships.

The mismatch is not just dimensional. Consider what a language model has internalized about its own embedding space. It has learned, through billions of gradient steps on text prediction, that certain directions in its embedding space correspond to grammatical agreement, others to sentiment polarity, others to named entity type, and so on. When an attention head in a language model processes a sequence of tokens, it interprets vectors within this well-structured space, using learned patterns to decide which tokens are relevant to which others. Feeding the language model a raw ViT patch embedding, without any transformation, would be like inserting a sentence in an unknown language into the middle of an English passage: the model would try to interpret the foreign tokens using English grammar rules, creating gibberish.

The projection layer must resolve both issues simultaneously: mapping from the vision embedding dimension $d_v$ to the language model dimension $d_l$, and transforming spatial-visual representations into semantic-linguistic representations that the LLM can interpret within its pre-trained context. This transformation is a technical requirement and a semantic necessity. The LLM has learned to reason about vectors in its input space, associating specific regions of that high-dimensional manifold with concepts. The projection layer must ensure that visual signals corresponding to these concepts land in the appropriate regions of the LLM's semantic space, allowing the model to use its linguistic knowledge for visual understanding.

A useful way to think about this is through the lens of coordinate systems. Imagine two experts, a cartographer and a navigator, each with a map of the same territory but using different coordinate systems. The cartographer uses latitude and longitude; the navigator uses bearing and distance. Both maps accurately represent the same geography, but without a conversion function, the navigator cannot use the cartographer's map. The projection layer is precisely this conversion function, a learned transformation that converts from the coordinate system of visual features to the coordinate system of linguistic representations. The underlying territory, the semantic concepts being represented, is the same in both spaces; only the coordinate conventions differ.

It is natural to wonder why projection should work at all. If vision encoders and language models are trained on completely different data with different objectives, why should their internal representations have any geometric relationship that a learned projection could exploit?

The answer lies in the shared semantic structure of the world. Both CLIP-style vision encoders and large language models are trained, directly or indirectly, on the internet, which is full of images with captions, articles with photographs, and visual media with textual context. Even if the same pixel data and the same text are not processed by both models simultaneously, the statistical regularities of the world ensure that both spaces develop clusters and directions corresponding to the same semantic categories. A dog looks like a dog whether you describe it in words or depict it in pixels.

More formally, consider a contrastive vision encoder like CLIP, trained to bring image embeddings close to their paired text embeddings. CLIP's vision encoder is explicitly trained to align with a text encoder, meaning its visual representations are already partly linguistic in character. When you use CLIP visual features as input to a projection layer targeting a separate LLM, you are building on partial alignment: CLIP's vision encoder has already been encouraged to organize its representations in a language-compatible way, making the projection's job substantially easier.

For vision encoders not trained contrastively, like those trained purely on image classification, the alignment is less direct but still exploitable. Classification training forces the encoder to represent visual categories, and categories correspond to words. The ImageNet classes, for instance, are linguistic labels, and training a vision encoder to classify into those labels creates representations that correlate with the language model's representations for those same category names. The projection layer, trained on enough image-text pairs, can learn to exploit these correlations even when the original encoders were trained independently.

The simplest and most widely used approach is linear projection, a single learnable matrix that affine-turns each vision embedding into the LLM's embedding space. This approach embodies the principle of Occam's razor: before introducing complexity, test whether a simple linear transformation suffices to align the two semantic spaces.

Given a vision encoder output $\mathbf{V} \in \mathbb{R}^{n \times d_v}$ where $n$ is the number of visual tokens (patches) and $d_v$ is the vision embedding dimension, the linear projection applies a weight matrix $\mathbf{W} \in \mathbb{R}^{d_v \times d_l}$ and optionally a bias $\mathbf{b} \in \mathbb{R}^{d_l}$:

where:

• $\mathbf{V} \in \mathbb{R}^{n \times d_v}$: the vision encoder output containing $n$ visual tokens (patches) of $d_v$ dimensions each
• $\mathbf{W} \in \mathbb{R}^{d_v \times d_l}$: the learnable weight matrix mapping from vision dimension $d_v$ to language model dimension $d_l$
• $\mathbf{b} \in \mathbb{R}^{d_l}$: the optional bias vector (often omitted in modern implementations)
• $\mathbf{1}_n$: a column vector of ones with length $n$, letting broadcasting of the bias to all $n$ tokens
• $\mathbf{H}_{\text{linear}} \in \mathbb{R}^{n \times d_l}$: the intermediate result after matrix multiplication, before bias addition
• $\mathbf{H} \in \mathbb{R}^{n \times d_l}$: the final projected visual tokens in the LLM's embedding space, ready for consumption by the language model

The matrix multiplication $\mathbf{V}\mathbf{W}$ performs the core geometric transformation. Each row of $\mathbf{V}$ is a single patch embedding, and right-multiplying by $\mathbf{W}$ maps that embedding into the LLM's embedding space. Geometrically, this rotates and scales while re-orienting the vision embedding space so that its directions align with meaningful directions in the LLM's space.

To understand the geometric action more concretely, recall from linear algebra that matrix multiplication is a linear map. A linear map can rotate vectors, stretch them along certain axes, compress them along others, and reflect them. It cannot, however, perform non-linear operations like folding the space or creating new clusters that do not exist in the original. This means linear projection can reorient the vision space to match the LLM's coordinate system, but it cannot fundamentally restructure the topology of the vision representations. If two visual concepts are indistinguishably close in vision space, a linear projection cannot separate them in LLM space. This is an inherent limitation we will revisit when discussing MLP projections.

In modern implementations, bias terms are often omitted following the convention established in transformer architectures, where Layer Normalization typically follows linear transformations, reducing the need for explicit bias. The projection applies independently to each visual token, meaning the sequence length $n$ remains unchanged; only the dimensionality transforms. This is an important property: the spatial correspondence between image regions and token positions is preserved exactly, which proves important for localized visual understanding tasks.

Linear projection relies on a key insight: if the vision encoder has learned semantically meaningful representations, a linear transformation may suffice to rotate and scale these representations into the LLM's embedding space. This assumes that the vision and language representations, while expressed in different coordinate systems, exist on approximately aligned semantic manifolds within their respective high-dimensional spaces.

To understand why this assumption often holds, consider how both vision encoders and language models learn their representations. Vision encoders trained via contrastive learning (like CLIP) or supervised classification learn to map visually similar images to nearby points in embedding space. Language models learn to map semantically similar words or contexts to nearby embeddings. When both models are trained on internet-scale data containing natural correlations between images and their descriptions, their respective embedding spaces often capture similar semantic relationships, albeit in different coordinate systems. A linear transformation can then serve as a coordinate rotation, aligning the axes of the vision space with those of the language space without changing the underlying semantic geometry.

The approach is computationally efficient, adding only $d_v \times d_l$ parameters, a negligible increase when the LLM itself contains billions of parameters. For a ViT-L/14 projected to LLaMA-2-7B, this is $1024 \times 4096 = 4{,}194{,}304$ parameters, roughly 0.06% the size of the LLM itself. The tiny parameter count makes overfitting unlikely even with modest training datasets and simplifies optimization considerably.

Linear projection also preserves spatial information faithfully since each patch embedding projects independently, maintaining the correspondence between image regions and token positions. This preservation of spatial structure proves important for tasks requiring localized understanding, such as referring expression comprehension (identifying which object in an image a phrase refers to) or visual question answering about specific image regions. When the language model's attention mechanism later processes these visual tokens, it can rely on their positional correspondence to build spatial reasoning chains.

Although linear projection has few parameters, how those parameters are initialized and trained matters considerably. Poor initialization can cause the projected visual features to lie in distant corners of the LLM's embedding space, where the language model has no useful pre-trained knowledge about vectors in that vicinity.

A common practice is to initialize the projection matrix with values drawn from a normal distribution scaled by $1 / \sqrt{d_v}$, similar to the initialization scheme used for transformer weight matrices. This keeps the initial output variance from growing or shrinking uncontrollably, helping the LLM receive visual tokens with magnitude comparable to its learned text token embeddings during early training.

Some implementations additionally use a learning rate for the projection layer that is several times larger than the learning rate for any unfrozen encoder parameters. Since the projection is learning from scratch while frozen encoders are simply being evaluated, the projection benefits from faster parameter updates. The asymmetric learning rate schedules help the projection converge while preventing instability in the frozen components.

While linear projection suffices for many applications, some architectures employ deeper projection networks, typically multi-layer perceptrons (MLPs), to capture non-linear relationships between vision and language spaces. These become particularly valuable when the pre-trained vision and language models originate from different training paradigms, or when the task requires fine-grained visual distinctions that must be remapped into complex linguistic descriptions.

An MLP projection introduces one or more hidden layers between the vision encoder and the LLM. The forward pass proceeds through these layers sequentially:

where:

• $\mathbf{V}$: the vision encoder output (as defined previously)
• $\mathbf{W}_1 \in \mathbb{R}^{d_v \times d_h}$: the first layer weight matrix mapping from vision dimension $d_v$ to hidden dimension $d_h$
• $\mathbf{b}_1 \in \mathbb{R}^{d_h}$: the first layer bias vector
• $\mathbf{h}_1 \in \mathbb{R}^{n \times d_h}$: the intermediate hidden representations after the non-linear activation
• $\mathbf{W}_2 \in \mathbb{R}^{d_h \times d_l}$: the second layer weight matrix mapping from hidden dimension $d_h$ to LLM dimension $d_l$
• $\mathbf{b}_2 \in \mathbb{R}^{d_l}$: the second layer bias vector
• $\mathbf{h}_2 \in \mathbb{R}^{n \times d_l}$: the final projected visual tokens

The hidden dimension $d_h$ is often chosen to be larger than both $d_v$ and $d_l$ to increase representational capacity. A common design expands to $4 \times d_v$ in the first layer, mirroring the expansion ratio used in standard transformer feed-forward networks, before compressing to $d_l$ in the second layer. This expansion creates a richer intermediate representation in which the non-linear activation can perform more expressive feature combinations.

The non-linear activation function, typically GELU or SiLU, is the key ingredient that distinguishes MLP from linear projection. Without activation functions, any stack of linear layers is mathematically equivalent to a single linear layer: $\mathbf{W}_2 (\mathbf{V} \mathbf{W}_1) = \mathbf{V}(\mathbf{W}_1 \mathbf{W}_2)$. The activation function breaks this collapse by introducing non-linearity between layers, letting the network to implement transformations that a single matrix could never represent.

What kinds of transformations require non-linearity? Consider a visual feature that encodes both "large" and "red" in a single direction in vision space, but the LLM represents "large" and "red" as separate, orthogonal directions in language space. A linear transformation cannot separate a single direction into two orthogonal ones; doing so requires non-linear disentanglement. The GELU activation, applied element-wise to the hidden layer, creates this capacity: different neurons can become selectively active based on input patterns, letting the network to effectively route different visual features to different linguistic dimensions.

MLP projections increase parameter count and computational cost during both training and inference. However, since projection occurs once per image (not per generated token), the overhead is amortized over long generation sequences. The additional capacity often improves performance on fine-grained visual understanding tasks where simple linear mappings struggle to translate subtle visual features into language-accessible representations.

The choice between linear and MLP projection involves balancing model capacity against optimization stability. Deeper projection networks can suffer from vanishing gradients or overfitting when training data is limited, and they may introduce unnecessary complexity if the vision and language spaces are already well-aligned. In practice, many state-of-the-art vision-language models, including early versions of LLaVA, successfully employ simple linear projections, suggesting that modern contrastive vision encoders (like CLIP) already produce representations sufficiently compatible with language model spaces to make non-linear transformations optional rather than mandatory.

A practical heuristic is to start with linear projection and upgrade to MLP only if benchmarks reveal that the model struggles with fine-grained distinctions or with adapting to a domain far from the vision encoder's pre-training distribution. For standard natural image understanding tasks using CLIP-based encoders, linear projection is often sufficient. For specialized domains, medical imaging, satellite data, or scientific charts, where the vision encoder's features may require substantial re-organization before they become linguistically useful, deeper projections are more likely to yield improvements.

LLaVA 1.5, for example, moved from the linear projection of the original LLaVA to an MLP projection (specifically a two-layer MLP with GELU activation) and reported improvements across visual question answering benchmarks. The authors attributed this to the MLP's ability to better align vision features with the LLM's embedding space, particularly for tasks requiring fine-grained visual understanding. The improvement was material but incremental, suggesting that the linear projection was already capturing the bulk of the necessary cross-modal alignment, with the MLP giving additional gains.

Both linear and MLP projections suffer from a scalability problem: they preserve the full sequence of visual tokens. For high-resolution images or vision encoders with small patch sizes, this can yield thousands of tokens, far exceeding the context limits of many LLMs and creating computational bottlenecks due to the quadratic complexity of attention.

Consider a high-resolution image of 896×896 pixels processed with a patch size of 14×14. This yields $(896/14)^2 = 4096$ patch tokens, which would consume an entire 4096-token context window without leaving any room for text instructions or generated responses. Even with standard 224×224 images and 16×16 patches, the 197 tokens required represent a significant fraction of typical context windows, limiting the length of conversations or the complexity of reasoning chains that can involve multiple images.

The Querying Transformer (Q-Former), introduced in BLIP-2, addresses this by learning to compress variable-length visual features into a fixed number of learnable query tokens. Rather than treating all visual tokens as equally important and passing them all to the LLM, the Q-Former learns to distill the visual information into a compact set of query representations that capture the most salient aspects of the image for language understanding.

The insight behind the Q-Former is that not all visual information is linguistically relevant. A detailed texture in the background, a subtle shadow on the ground, or the precise number of leaves on a tree may be visually present but rarely matters for answering questions or generating captions. Rather than forcing the LLM to sift through hundreds of visual tokens to find the few relevant ones, the Q-Former does this filtering proactively, presenting the LLM with a compressed summary that retains what matters for language-based tasks.

The Q-Former consists of a lightweight transformer, typically BERT-base sized with around 188M parameters, positioned between the frozen vision encoder and the frozen LLM. It employs two attention mechanisms:

1. Self-attention among queries: The learnable query tokens attend to each other, letting them to coordinate and avoid redundancy
2. Cross-attention to vision features: Queries attend to the vision encoder outputs, extracting and compressing visual information

The key innovation lies in the queries themselves: a fixed set of learnable embeddings $\mathbf{Q} \in \mathbb{R}^{m \times d_q}$ where $m$ (typically 32) is a hyperparameter much smaller than the number of visual tokens. These queries attend to the vision features through cross-attention layers, effectively learning to extract and compress the most relevant visual information.

This architecture functions analogously to a database query system. Instead of presenting the LLM with the entire raw dataset (all visual tokens), the Q-Former acts as an intelligent query processor that extracts specific information based on learnable query templates. Each of the $m$ learnable queries can be thought of as a specialized probe, trained to extract particular types of visual information: global scene context, specific object categories, spatial relationships, fine-grained textures, or action cues. Through training on image-text pairs, these queries learn which aspects of the visual input are most relevant for generating accurate and informative text responses.

The fact that the query vectors are learnable parameters, not computed from the input, is important. Unlike in standard attention where queries are derived from the input sequence, Q-Former queries are persistent learned embeddings that carry their function across all images. Query 3 might always look for foreground objects; Query 15 might always probe for background context. This specialization emerges from training and enables efficient extraction without requiring each image to re-derive the extraction strategy from scratch.

The Q-Former operates similarly to the standard transformer but with a modified attention pattern. For a given layer, the self-attention over queries follows the standard scaled dot-product, letting the learnable queries to exchange information among themselves before probing the visual features:

where:

• $\mathbf{Q} \in \mathbb{R}^{m \times d_q}$: the learnable query embeddings with $m$ queries of dimension $d_q$
• $\mathbf{W}_Q, \mathbf{W}_K \in \mathbb{R}^{d_q \times d_k}$: the query and key projection matrices mapping to key dimension $d_k$
• $\mathbf{W}_V \in \mathbb{R}^{d_q \times d_q}$: the value projection matrix (typically $d_q$ in standard configurations)
• $d_k$: the dimension of the key vectors, used to scale the dot products and prevent softmax saturation at high dimensionality

The softmax is applied row-wise over the attention scores. This produces a distribution over the $m$ queries. This self-attention step allows the query tokens to share context and coordinate their strategies before extracting information from the vision features. This keeps subsequent cross-attention extracts complementary rather than redundant information. If several queries all attend to the same visual region and extract the same features, the compression is wasted; self-attention helps the queries diversify their coverage.

The cross-attention mechanism allows queries to attend to vision features $\mathbf{V}$:

where:

• $\mathbf{Q}$: the learnable query embeddings (as defined previously), serving as the query source
• $\mathbf{V} \in \mathbb{R}^{n \times d_v}$: the vision encoder output with $n$ visual tokens of dimension $d_v$, serving as the key-value source
• $\mathbf{W}_Q \in \mathbb{R}^{d_q \times d_k}$: the query projection matrix for the learnable queries
• $\mathbf{W}_K, \mathbf{W}_V \in \mathbb{R}^{d_v \times d_k}$: the key and value projection matrices that transform vision features into the attention computation's space

The softmax is computed over the $n$ vision tokens, letting each of the $m$ queries to attend to all visual patches. The output has shape $[m, d_k]$, compressing the variable number of vision tokens $n$ into the fixed number of query tokens $m$. This is the basic compression step: attention-weighted pooling of vision features, where the weights are determined by the learned queries.

The vision features $\mathbf{V}$ serve as keys and values but do not attend to the queries. The information flow is strictly one-directional: queries pull information from the vision features, but the vision features remain static. This asymmetry enables compression by forcing the queries to extract information, rather than allowing bidirectional flow that would maintain sequence length. It also means the Q-Former cannot modify or distort the vision features themselves, preserving the integrity of the visual representation while creating a compact summary.

The output of the Q-Former is a sequence of $m$ query embeddings, which then undergo a final linear projection into the LLM's input dimension. This compresses, for example, 197 ViT tokens into 32 query tokens, a 6x reduction in sequence length that allows the LLM to process multiple images or engage in much longer conversations without exhausting its context window.

The Q-Former employs a carefully designed multi-stage training procedure to align its queries with both the vision encoder and the LLM. This staged approach is necessary because the Q-Former must simultaneously learn two very different things: what visual information is salient for language understanding, and how to format that information for consumption by a specific language model.

The three training stages are:

Stage 1: Vision-Language Representation Learning. The Q-Former, paired with a frozen vision encoder, is trained on image-text pairs using three objectives simultaneously: image-text contrastive learning (bringing image and text representations close), image-grounded text generation (generating captions from visual features), and image-text matching (predicting whether an image and text are paired). These three objectives teach the queries different aspects of visual-linguistic alignment. Contrastive learning trains the queries to extract global semantic content that discriminates between different images. Text generation trains the queries to preserve detail needed for accurate description. Matching trains the queries to capture coherence across visual content and linguistic descriptions.

Stage 2: Bootstrap Vision-to-Language Generation. With the Q-Former's output now semantically meaningful, a frozen LLM is introduced. A lightweight linear projection maps Q-Former outputs to the LLM's embedding dimension, and the system is trained with a standard language modeling objective: predicting the next token in a caption given the compressed visual tokens. This stage trains the projection between Q-Former and LLM while freezing both the Q-Former (whose queries were just trained) and the LLM (whose parameters we want to preserve). The Q-Former is a learned "soft token" generator, and the projection layer learns to format those soft tokens in the LLM's native embedding space.

Stage 3: Task-Specific Fine-tuning. The full system is adapted to specific downstream tasks. Depending on the task and computational budget, various combinations of components may be unfrozen.

This staged approach ensures that the queries learn to extract visual information relevant to language understanding before attempting to interface with a specific LLM. The first stage teaches the queries what visual information is semantically salient, while the second stage teaches how to format that information for a specific language model's embedding space. The bootstrapping idea is particularly clever: rather than trying to train the Q-Former and projection simultaneously from scratch with respect to a frozen LLM, which creates an extremely difficult optimization problem, the staged approach first establishes a useful intermediate representation and then teaches the final projection step.

Training the projection layer involves decisions about what components to freeze and how to structure the optimization process. These choices significantly impact computational cost, training stability, and final performance. The basic tension lies between preserving the capabilities of pre-trained models and adapting them to work together effectively.

Modern vision-language models typically employ one of three strategies:

Vision Frozen, LLM Frozen, Projection Trainable. This is the most common approach, exemplified by LLaVA and BLIP-2. The vision encoder and LLM retain their pre-trained weights, while only the projection parameters update. This preserves the capabilities of both pre-trained models while learning the interface between them. It requires the least compute and memory but limits the system's ability to adapt vision features to the specific demands of language understanding.

This freezing strategy rests on the assumption that both the vision encoder and language model have already learned general, transferable representations from their respective pre-training phases. By keeping them frozen, we prevent catastrophic forgetting, the phenomenon where a model loses previously learned capabilities when trained on new tasks. Since vision encoders like CLIP were trained on diverse image-text pairs and language models like LLaMA were trained on broad text corpora, freezing them maintains these general capabilities while the projection layer learns the specific translations needed for cross-modal interaction. The result is a system where the projection layer bears the entire burden of bridging the modalities, which can work well precisely because the pre-trained representations already have much of the semantic structure needed for the bridge to hold.

Vision Trainable, LLM Frozen. Sometimes used when the vision encoder is relatively small, or when domain adaptation is important. This allows visual representations to shift toward the language space but risks catastrophic forgetting of visual pre-training. If the vision encoder was pre-trained on natural images but the application involves medical imagery or satellite photos, unfreezing the vision encoder enables it to adapt its feature extraction to the new domain. The projection layer, now receiving domain-adapted visual features, can learn a cleaner mapping to the language space. However, this strategy requires careful regularization to prevent the vision encoder from drifting too far from its original representations, which could destabilize the projection.

Full Fine-tuning. Unfreezing all components often yields the best performance on specific tasks but requires massive compute and risks destabilizing either the vision or language representations. When the vision-language task involves specialized domains with distinct visual and linguistic patterns, such as scientific diagram understanding or technical document analysis, full fine-tuning allows the entire system to co-adapt. This approach demands significant GPU resources and careful tuning of learning rates across different components, as vision encoders and language models may require different optimization dynamics: vision encoders typically benefit from smaller learning rates to preserve their pre-trained visual features, while projection layers trained from scratch can tolerate larger learning rates to converge faster.

Many successful implementations use a two-stage approach to train projection layers effectively.

Stage 1 (Feature Alignment). Train the projection on large-scale image-text pairs with simple objectives, typically contrastive learning or image captioning. This stage establishes coarse alignment between modalities. The projection learns basic associations between visual concepts and their linguistic descriptions. For instance, visual features corresponding to furry animals, four legs, and pointed ears should map to linguistic embeddings near the word "dog." This alignment phase typically uses large, noisy datasets like LAION-400M or Conceptual Captions, where the abundance of data compensates for the simplicity of the training objective.

The first stage is often quick relative to later stages. Because the objective is simple and the model has no complex instruction-following behavior to learn, the projection can establish coarse alignment within a few hours on standard GPU hardware. This coarse alignment is sufficient to proceed to the second stage, where fine-grained adjustments take place.

Stage 2 (Instruction Tuning). Fine-tune on instruction-following data with the LLM's language modeling objective, adapting the projection for conversational and reasoning capabilities. As we discussed in the instruction tuning chapter, this second stage proves important for letting helpful assistant behaviors beyond simple captioning. The projection layer must learn to translate visual features into static descriptions and to present visual information in ways that support multi-turn reasoning, following complex instructions, and engaging in dialogue about image content.

The instruction tuning stage uses carefully curated datasets of image-question-answer triplets or multi-turn visual conversations. LLaVA used GPT-4 to generate such instruction-following data from image captions and bounding boxes, creating a dataset of around 150,000 examples that taught the model to follow instructions about visual content. During this stage, the projection learns which visual features correspond to which linguistic concepts and how to retrieve the relevant visual features given the context of a user's question.

When training with conversational data, it is needed to mask the loss on visual tokens and compute gradients only on text tokens. The projection layer receives gradients during text generation, learning to produce visual representations that facilitate accurate captioning or response generation, but we do not train the system to "reconstruct" images from text or to predict visual token values.

This masking strategy reflects the asymmetric nature of vision-language generation. During training, when the model processes a sequence like [visual tokens][text tokens: "A dog playing in the park"], the loss is computed only on predicting the text tokens "A dog playing in the park" given the visual tokens and any preceding text. The projection layer receives gradients flowing back from these text prediction losses, updating its weights to produce visual representations that better support accurate text generation, but no loss is applied to the visual tokens themselves.

Without loss masking, the model might be penalized for the raw numerical values of its visual token representations, which would create a misleading training signal: we want the visual tokens to be useful for generating text, not to take specific numerical values. The masking ensures that the only feedback the projection receives is whether the LLM can accurately generate the corresponding text given the visual representations, which is precisely the objective we care about.

Consider building a vision-language model using CLIP ViT-L/14 as the vision encoder and LLaMA-2-7B as the language model. This combination represents a common architectural choice in open-source vision-language models, combining the strong zero-shot visual capabilities of CLIP with the reliable reasoning abilities of LLaMA.

Vision Encoder Output:

The ViT-L/14 processes a 224×224 pixel image by dividing it into 14×14 patches. The number of patch tokens is $(224/14)^2 = 256$. Adding the class token yields 257 visual tokens in total. Each token is a 1024-dimensional vector, giving a vision encoder output tensor of shape $[257, 1024]$.

The choice of ViT-L/14 provides a good balance between computational efficiency and representational capacity. The 14×14 patch size offers finer spatial resolution than the more common 16×16, capturing more detail in the visual features, while the 1024-dimensional embeddings provide rich representations for each patch.

Target LLM Input:

LLaMA-2-7B uses a 4096-dimensional embedding space. The projection must transform each 1024-dimensional vision vector into a 4096-dimensional LLM vector, an expansion by a factor of four.

Linear Projection:

The weight matrix has shape $1024 \times 4096$, containing $4{,}194{,}304$ parameters. For each of the 257 visual tokens, we compute:

The output tensor has shape $[257, 4096]$: 257 visual tokens, each now in LLaMA's 4096-dimensional space. These 257 tokens are then prepended to the text token embeddings, and the full sequence is fed into LLaMA-2-7B for generation. The 257 visual tokens represent 6.3% of LLaMA's 4096-token context window, leaving 3839 tokens for text.

The linear projection adds only 4.2 million parameters, a tiny fraction of the billions in the LLM. Despite its simplicity, this projection must perform the important mapping from CLIP's contrastive visual space to LLaMA's autoregressive linguistic space.

Q-Former Compression:

The Q-Former uses 32 learnable queries, each of dimension 768. The cross-attention compresses the 257 input tokens down to 32 output tokens of dimension 768. A final linear projection maps these 32 vectors from 768 dimensions to 4096 dimensions (LLaMA's space), adding $768 \times 4096 = 3{,}145{,}728$ additional parameters.

Using Q-Former, 257 visual tokens that would occupy 6.3% of the context window become only 32 tokens, leaving 4064 positions for text. Processing two images with linear projection would require 514 tokens (12.5% of the context), while Q-Former would require only 64 tokens (1.6%). For multi-image or multi-turn visual conversations, this difference in context efficiency is decisive.

The tradeoff is information compression. The Q-Former must decide which of the 257 input patches' information to retain in its 32 output tokens. For simple image captioning, 32 tokens may be sufficient to capture the key objects and scene context. For tasks requiring precise localization or counting, the compression may lose fine-grained spatial detail that the language model would have needed.

Let us implement the three projection approaches in PyTorch, building a modular vision-language projection system.

The text_labels tensor with -100 fill values is the key implementation detail. PyTorch's CrossEntropyLoss ignores positions where the target is -100, so the visual token positions contribute zero gradient and zero loss. The projection layer learns entirely from the text generation loss on the positions that follow the visual tokens, exactly the signal we want.

The key parameters for vision-language projection control the basic capacity and efficiency tradeoffs of the cross-modal bridge:

• vision_dim: The dimension of the vision encoder output (e.g., 1024 for ViT-L/14). This determines the input size to the projection layer. Larger vision models like ViT-H/14 (1280 dimensions) or ViT-G (1664 dimensions) provide richer visual features but require larger projection matrices. The choice of vision encoder is often the most consequential design decision for the quality of visual understanding.

• llm_dim: The dimension of the language model embedding space (e.g., 4096 for LLaMA-2-7B, 8192 for LLaMA-70B). The projection output must match this dimension exactly. Larger language models typically have larger embedding dimensions, requiring more projection parameters. The projection must learn to fill a higher-dimensional space with meaningful visual information, which generally benefits from more expressive projection architectures.

• bias: Whether to include bias terms in linear projections (typically False in modern implementations). When the projection is followed by Layer Normalization (which is common), the bias term is redundant since LayerNorm has its own learnable shift parameter. Omitting the bias reduces the parameter count slightly and can improve optimization stability.

• hidden_dim: For MLP projections, the intermediate dimension (often $4 \times$ vision_dim) that increases representational capacity. The expansion ratio of four mirrors the standard transformer feed-forward network design. Increasing this ratio allows more complex non-linear transformations but increases both parameter count and the risk of overfitting.

• num_queries: For Q-Former, the number of learnable query tokens (typically 32) that compress variable-length visual features into a fixed sequence. Increasing this number preserves more visual detail and improves performance on tasks requiring fine-grained spatial understanding, but reduces the compression benefit and increases the visual token budget consumed from the LLM's context window.

• qformer_dim: The hidden dimension for Q-Former transformer layers (e.g., 768, matching BERT-base). This controls the expressiveness of the cross-attention mechanism and the information capacity of each query token. The BLIP-2 authors used 768 to reuse BERT-base weights for initialization, accelerating convergence.

• qformer_heads: The number of attention heads in Q-Former layers, letting parallel extraction of different aspects of visual information. Standard configurations use 12 heads matching BERT-base, though the optimal count depends on the dimensionality and the diversity of visual information that must be captured.

• qformer_layers: The depth of the Q-Former transformer, controlling the iterative refinement of query representations. More layers allow queries to progressively integrate information across the image, with early layers capturing local features and later layers building global context, at the cost of additional computation and training time.

The choice of projection mechanism involves basic tradeoffs between computational efficiency, information preservation, and training stability. Each approach has characteristic failure modes that become apparent in practice.

Linear projections, while efficient and easy to optimize, create a potential information bottleneck when vision and language embedding spaces are significantly misaligned. If the vision encoder represents "redness" and "roundness" as independent, orthogonal directions in its space while the language model encodes these properties in a correlated subspace, a linear transformation cannot reconcile this structural difference without encountering geometric conflicts. The linear mapping can rotate and scale, but it cannot create new distinctions or collapse distinctions that exist in the source space. For most natural image understanding tasks with CLIP-based encoders, this limitation is rarely binding because CLIP was trained contrastively to align with language. But for specialized domains, the rigid linearity can become a meaningful constraint.

Linear projections also preserve the full sequence length of visual tokens, which becomes prohibitively expensive for high-resolution images. A 448×448 image with 14×14 patches yields 1024 tokens, consuming a full quarter of LLaMA's 4096-token context window before any text is added. Processing two such images would leave almost no room for instructions or generated responses. This limitation is not merely theoretical: production multimodal systems routinely encounter long conversations or multi-image inputs where context budget is a real constraint.

MLP projections offer greater representational capacity but introduce additional parameters and potential optimization challenges. Deep projection networks can suffer from initialization sensitivity when trained alongside frozen encoders. The gradient signal must flow from the LLM's loss through the LLM's embedding layer, through the projection, back to the frozen vision encoder outputs, and this chain can attenuate or distort unless learning rates and initializations are carefully chosen. The non-linearities, while powerful, create optimization surfaces with multiple local minima, making the training more sensitive to hyperparameter choices than linear projection. An unfortunate local minimum can produce a projection that maps all visual inputs to similar LLM embeddings, effectively causing the model to ignore image content.

The Q-Former approach addresses the sequence length problem through compression but introduces architectural complexity and risks information loss. The basic tension is that 32 query tokens must capture everything the LLM needs to know about an image that contains potentially hundreds of distinct visual elements. For a scene with twenty distinct objects, each query must multiplex information about multiple entities, potentially leading to confusion when the LLM tries to ask about specific objects. Tasks requiring precise spatial localization, fine-grained counting, or detailed texture description are particularly vulnerable to this compression.

The Q-Former also requires substantially more training to converge than simpler projections. Its three-stage training procedure is computationally expensive, and the cross-attention mechanism involves more parameters and more complex gradients than a simple linear projection. For applications where training compute is limited, the Q-Former's potential quality advantages may not justify its training cost compared to a simple linear projection on more data.

A deeper limitation, shared by all projection approaches, is the assumption that the LLM can successfully interpret visual tokens as if they were linguistic tokens. The LLM was trained exclusively on text; its attention mechanism, layer normalization statistics, and feed-forward weights are all calibrated to text-like distributions. Visual tokens, even after projection, may violate these assumptions in subtle ways. For instance, visual tokens representing spatial patches of a smooth gradient may all be nearly identical, violating the assumption of token diversity that language models are accustomed to. Research into specialized normalization strategies for visual tokens, or hybrid training objectives that expose the LLM to visual token statistics during pre-training, addresses this assumption but remains an active area of investigation.

Despite these limitations, projection layers have enabled the current generation of multimodal AI systems. They allow us to compose pre-trained components, combining the visual understanding of CLIP with the reasoning capabilities of LLaMA, without requiring end-to-end training of massive models from scratch. This modular approach democratizes vision-language research, letting academic labs and small teams to build competitive models by combining open-source vision encoders with language models. Without projection layers, every new multimodal capability would require re-training the entire stack from scratch, which is feasible only for a handful of organizations globally.

The practical impact of this modular approach extends beyond research. It enables rapid domain adaptation: a medical imaging company can take a general-purpose vision encoder fine-tuned on radiology images and connect it to a general language model through a projection layer trained on radiology report data, achieving specialized multimodal capability at a fraction of the cost of training a full multimodal model from scratch. It enables model swapping: as better vision encoders or language models become available, they can be plugged into an existing architecture with only the projection layer requiring re-training.

Looking ahead, as we explore architectures like LLaVA and Flamingo in upcoming chapters, we will see how these projection techniques serve as the foundation for systems capable of engaging in visual question answering, image captioning, and multimodal dialogue. The projection layer, whether a simple matrix multiplication or a query transformer, remains the bridge letting machines to see and speak simultaneously. Future research directions include adaptive projection mechanisms that adjust compression ratios based on image complexity, dynamic query generation that creates task-specific queries rather than using fixed learned queries, and cross-modal pre-training strategies that expose LLMs to visual token distributions during pre-training, reducing the semantic gap that projection layers must bridge.

Vision-language projection solves the basic architectural challenge of connecting two pre-trained models that were trained in isolation on entirely different modalities. The three main approaches represent a progression from simple to expressive:

Linear projection uses a single weight matrix to map each vision token from the vision encoder's dimension to the LLM's dimension. It is parameter-efficient (roughly 4 million parameters for a ViT-L/14 to LLaMA-2-7B bridge), preserves spatial structure faithfully, and optimizes cleanly. It works well when the vision encoder was trained contrastively and its representations are already partially aligned with language. Its principal limitation is that it preserves the full visual token sequence, which can exhaust context budgets for high-resolution or multi-image inputs.

MLP projection introduces non-linear transformations through hidden layers and activation functions, letting the projection to learn more complex feature re-organizations that pure linear transformations cannot represent. This proves valuable when vision and language encoders originate from very different training paradigms, or when tasks require fine-grained visual distinctions that need non-linear mapping into language space. The cost is increased parameter count and greater sensitivity to initialization and learning rate choices.

Q-Former addresses the token budget problem through attention-based compression. A set of fixed learnable query embeddings attend to the visual tokens via cross-attention, extracting a fixed-size summary regardless of how many input patches the vision encoder produces. The 6x-8x compression (from 197-257 tokens down to 32) frees substantial context window capacity, letting longer conversations and multi-image inputs at the cost of potential fine-grained detail loss and substantially higher training complexity.

Training strategies complement the architectural choices. Freezing both the vision encoder and LLM while training only the projection is the standard baseline: it preserves pre-trained capabilities, prevents catastrophic forgetting, and minimizes compute requirements. Stage-wise training, first aligning modalities on large noisy datasets and then fine-tuning for instruction-following on curated data, produces projections that both accurately translate visual concepts and effectively support conversational reasoning. Loss masking on visual token positions ensures the projection learns to produce representations useful for text generation, not to reconstruct input features.

Together, these projection mechanisms have made the current generation of multimodal AI systems practical, letting the composition of specialized pre-trained models into unified vision-language systems capable of reasoning about visual content without requiring end-to-end training from scratch.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about vision-language projection mechanisms.