Vision-Language Projection › MLP Projection

While linear projection suffices for many applications, some architectures employ deeper projection networks, typically multi-layer perceptrons (MLPs), to capture non-linear relationships between vision and language spaces. These become particularly valuable when the pre-trained vision and language models originate from different training paradigms, or when the task requires fine-grained visual distinctions that must be remapped into complex linguistic descriptions.

An MLP projection introduces one or more hidden layers between the vision encoder and the LLM. The forward pass proceeds through these layers sequentially:

$$\begin{aligned} \mathbf{h}_1 &= \text{GELU}(\mathbf{V} \mathbf{W}_1 + \mathbf{b}_1) \\ \mathbf{h}_2 &= \mathbf{h}_1 \mathbf{W}_2 + \mathbf{b}_2 \end{aligned}$$

where:

• $\mathbf{V}$: the vision encoder output (as defined previously)
• $\mathbf{W}_1 \in \mathbb{R}^{d_v \times d_h}$: the first layer weight matrix mapping from vision dimension $d_v$ to hidden dimension $d_h$
• $\mathbf{b}_1 \in \mathbb{R}^{d_h}$: the first layer bias vector
• $\mathbf{h}_1 \in \mathbb{R}^{n \times d_h}$: the intermediate hidden representations after the non-linear activation
• $\mathbf{W}_2 \in \mathbb{R}^{d_h \times d_l}$: the second layer weight matrix mapping from hidden dimension $d_h$ to LLM dimension $d_l$
• $\mathbf{b}_2 \in \mathbb{R}^{d_l}$: the second layer bias vector
• $\mathbf{h}_2 \in \mathbb{R}^{n \times d_l}$: the final projected visual tokens

The hidden dimension $d_h$ is often chosen to be larger than both $d_v$ and $d_l$ to increase representational capacity. A common design expands to $4 \times d_v$ in the first layer, mirroring the expansion ratio used in standard transformer feed-forward networks, before compressing to $d_l$ in the second layer. This expansion creates a richer intermediate representation in which the non-linear activation can perform more expressive feature combinations.

The non-linear activation function, typically GELU or SiLU, is the key ingredient that distinguishes MLP from linear projection. Without activation functions, any stack of linear layers is mathematically equivalent to a single linear layer: $\mathbf{W}_2 (\mathbf{V} \mathbf{W}_1) = \mathbf{V}(\mathbf{W}_1 \mathbf{W}_2)$. The activation function breaks this collapse by introducing non-linearity between layers, letting the network to implement transformations that a single matrix could never represent.

What kinds of transformations require non-linearity? Consider a visual feature that encodes both "large" and "red" in a single direction in vision space, but the LLM represents "large" and "red" as separate, orthogonal directions in language space. A linear transformation cannot separate a single direction into two orthogonal ones; doing so requires non-linear disentanglement. The GELU activation, applied element-wise to the hidden layer, creates this capacity: different neurons can become selectively active based on input patterns, letting the network to effectively route different visual features to different linguistic dimensions.

MLP projections increase parameter count and computational cost during both training and inference. However, since projection occurs once per image (not per generated token), the overhead is amortized over long generation sequences. The additional capacity often improves performance on fine-grained visual understanding tasks where simple linear mappings struggle to translate subtle visual features into language-accessible representations.

The choice between linear and MLP projection involves balancing model capacity against optimization stability. Deeper projection networks can suffer from vanishing gradients or overfitting when training data is limited, and they may introduce unnecessary complexity if the vision and language spaces are already well-aligned. In practice, many state-of-the-art vision-language models, including early versions of LLaVA, successfully employ simple linear projections, suggesting that modern contrastive vision encoders (like CLIP) already produce representations sufficiently compatible with language model spaces to make non-linear transformations optional rather than mandatory.

A practical heuristic is to start with linear projection and upgrade to MLP only if benchmarks reveal that the model struggles with fine-grained distinctions or with adapting to a domain far from the vision encoder's pre-training distribution. For standard natural image understanding tasks using CLIP-based encoders, linear projection is often sufficient. For specialized domains, medical imaging, satellite data, or scientific charts, where the vision encoder's features may require substantial re-organization before they become linguistically useful, deeper projections are more likely to yield improvements.

LLaVA 1.5, for example, moved from the linear projection of the original LLaVA to an MLP projection (specifically a two-layer MLP with GELU activation) and reported improvements across visual question answering benchmarks. The authors attributed this to the MLP's ability to better align vision features with the LLM's embedding space, particularly for tasks requiring fine-grained visual understanding. The improvement was material but incremental, suggesting that the linear projection was already capturing the bulk of the necessary cross-modal alignment, with the MLP giving additional gains.