Vision-Language Projection › Linear Projection

The simplest and most widely used approach is linear projection, a single learnable matrix that affine-turns each vision embedding into the LLM's embedding space. This approach embodies the principle of Occam's razor: before introducing complexity, test whether a simple linear transformation suffices to align the two semantic spaces.

Given a vision encoder output $\mathbf{V} \in \mathbb{R}^{n \times d_v}$ where $n$ is the number of visual tokens (patches) and $d_v$ is the vision embedding dimension, the linear projection applies a weight matrix $\mathbf{W} \in \mathbb{R}^{d_v \times d_l}$ and optionally a bias $\mathbf{b} \in \mathbb{R}^{d_l}$:

$$\begin{aligned} \mathbf{H}_{\text{linear}} &= \mathbf{V} \mathbf{W} \\ \mathbf{H} &= \mathbf{H}_{\text{linear}} + \mathbf{1}_n \mathbf{b}^\top \end{aligned}$$

where:

• $\mathbf{V} \in \mathbb{R}^{n \times d_v}$: the vision encoder output containing $n$ visual tokens (patches) of $d_v$ dimensions each
• $\mathbf{W} \in \mathbb{R}^{d_v \times d_l}$: the learnable weight matrix mapping from vision dimension $d_v$ to language model dimension $d_l$
• $\mathbf{b} \in \mathbb{R}^{d_l}$: the optional bias vector (often omitted in modern implementations)
• $\mathbf{1}_n$: a column vector of ones with length $n$, letting broadcasting of the bias to all $n$ tokens
• $\mathbf{H}_{\text{linear}} \in \mathbb{R}^{n \times d_l}$: the intermediate result after matrix multiplication, before bias addition
• $\mathbf{H} \in \mathbb{R}^{n \times d_l}$: the final projected visual tokens in the LLM's embedding space, ready for consumption by the language model

The matrix multiplication $\mathbf{V}\mathbf{W}$ performs the core geometric transformation. Each row of $\mathbf{V}$ is a single patch embedding, and right-multiplying by $\mathbf{W}$ maps that embedding into the LLM's embedding space. Geometrically, this rotates and scales while re-orienting the vision embedding space so that its directions align with meaningful directions in the LLM's space.

To understand the geometric action more concretely, recall from linear algebra that matrix multiplication is a linear map. A linear map can rotate vectors, stretch them along certain axes, compress them along others, and reflect them. It cannot, however, perform non-linear operations like folding the space or creating new clusters that do not exist in the original. This means linear projection can reorient the vision space to match the LLM's coordinate system, but it cannot fundamentally restructure the topology of the vision representations. If two visual concepts are indistinguishably close in vision space, a linear projection cannot separate them in LLM space. This is an inherent limitation we will revisit when discussing MLP projections.

In modern implementations, bias terms are often omitted following the convention established in transformer architectures, where Layer Normalization typically follows linear transformations, reducing the need for explicit bias. The projection applies independently to each visual token, meaning the sequence length $n$ remains unchanged; only the dimensionality transforms. This is an important property: the spatial correspondence between image regions and token positions is preserved exactly, which proves important for localized visual understanding tasks.

Linear projection relies on a key insight: if the vision encoder has learned semantically meaningful representations, a linear transformation may suffice to rotate and scale these representations into the LLM's embedding space. This assumes that the vision and language representations, while expressed in different coordinate systems, exist on approximately aligned semantic manifolds within their respective high-dimensional spaces.

To understand why this assumption often holds, consider how both vision encoders and language models learn their representations. Vision encoders trained via contrastive learning (like CLIP) or supervised classification learn to map visually similar images to nearby points in embedding space. Language models learn to map semantically similar words or contexts to nearby embeddings. When both models are trained on internet-scale data containing natural correlations between images and their descriptions, their respective embedding spaces often capture similar semantic relationships, albeit in different coordinate systems. A linear transformation can then serve as a coordinate rotation, aligning the axes of the vision space with those of the language space without changing the underlying semantic geometry.

The approach is computationally efficient, adding only $d_v \times d_l$ parameters, a negligible increase when the LLM itself contains billions of parameters. For a ViT-L/14 projected to LLaMA-2-7B, this is $1024 \times 4096 = 4{,}194{,}304$ parameters, roughly 0.06% the size of the LLM itself. The tiny parameter count makes overfitting unlikely even with modest training datasets and simplifies optimization considerably.

Linear projection also preserves spatial information faithfully since each patch embedding projects independently, maintaining the correspondence between image regions and token positions. This preservation of spatial structure proves important for tasks requiring localized understanding, such as referring expression comprehension (identifying which object in an image a phrase refers to) or visual question answering about specific image regions. When the language model's attention mechanism later processes these visual tokens, it can rely on their positional correspondence to build spatial reasoning chains.

Although linear projection has few parameters, how those parameters are initialized and trained matters considerably. Poor initialization can cause the projected visual features to lie in distant corners of the LLM's embedding space, where the language model has no useful pre-trained knowledge about vectors in that vicinity.

A common practice is to initialize the projection matrix with values drawn from a normal distribution scaled by $1 / \sqrt{d_v}$, similar to the initialization scheme used for transformer weight matrices. This keeps the initial output variance from growing or shrinking uncontrollably, helping the LLM receive visual tokens with magnitude comparable to its learned text token embeddings during early training.

Some implementations additionally use a learning rate for the projection layer that is several times larger than the learning rate for any unfrozen encoder parameters. Since the projection is learning from scratch while frozen encoders are simply being evaluated, the projection benefits from faster parameter updates. The asymmetric learning rate schedules help the projection converge while preventing instability in the frozen components.