LLaVA Architecture › Architectural Components

LLaVA's architecture consists of three basic components working in concert. Understanding each piece and their interaction reveals why this design has become the template for modern open-source multimodal assistants. The elegance of this tripartite structure lies in its modularity: each component can be upgraded or modified independently, letting the architecture to evolve as better vision encoders or language models become available without requiring basic architectural changes.

LLaVA uses a pre-trained CLIP visual encoder, specifically the Vision Transformer (ViT) architecture. As discussed in prior chapters on Vision Transformers and CLIP, this encoder processes input images through patch embeddings and transformer layers to produce a sequence of visual feature vectors. The choice of CLIP is particularly strategic because CLIP has already learned to align visual features with linguistic concepts through its contrastive training objective. This means the vision encoder does not merely extract generic visual features; it produces representations that are already grounded in semantic meaning, making the subsequent alignment with the language model materially more tractable.

These visual tokens capture hierarchical visual information, from edges and textures in early layers to semantic concepts like objects and scenes in deeper layers. The transformer architecture processes these patches through self-attention mechanisms, letting each visual token to incorporate context from the entire image. This global receptive field means that even individual patch embeddings contain information about the full image content, not just their local region. A patch representing the corner of a room, for example, will have attended to the patches showing furniture and lighting as well as wall color, so its embedding implicitly encodes information about the room type and style.

LLaVA keeps this encoder frozen throughout training, preserving the rich visual representations learned from CLIP's extensive pre-training on image-text pairs. Freezing prevents catastrophic forgetting of the visual concepts CLIP has learned and maintains the stability of the visual feature distribution, which is needed for reliable multimodal alignment. If the vision encoder were allowed to drift during LLaVA training, the projection layer would be chasing a moving target, learning to project features from a shifting distribution rather than from a stable, well-understood encoder.

Another reason to freeze the vision encoder is parameter efficiency. The CLIP ViT-L/14 encoder contains roughly 307 million parameters. Allowing all of these to update during multimodal training would require materially more data and compute to avoid overfitting to the relatively small instruction tuning dataset. Freezing reduces the effective training problem to a much smaller parameter set, making convergence faster and more predictable.

The central innovation of LLaVA is the connector bridging vision and language modalities. Rather than employing complex cross-attention mechanisms, custom fusion layers, or detailed gating mechanisms that would require substantial architectural modifications to the language model, LLaVA uses a straightforward linear transformation (or multi-layer perceptron in later versions) to map visual features into the word embedding space of the language model. This simplicity is deliberate and powerful: it treats the alignment problem as a straightforward translation task between two fixed representation spaces.

Given vision features $Z \in \mathbb{R}^{N \times d_v}$, the projection layer produces language embeddings by learning a linear transformation from the vision feature space to the LLM's embedding space:

$$H_v = Z \cdot W \in \mathbb{R}^{N \times d_{\text{LLM}}}$$

where:

• $H_v$: the projected visual embeddings in the LLM's input space (shape $N \times d_{\text{LLM}}$)
• $Z$: the visual feature matrix from the frozen vision encoder (shape $N \times d_v$)
• $W$: the trainable projection matrix mapping vision to language space (shape $d_v \times d_{\text{LLM}}$)
• $d_{\text{LLM}}$: the hidden dimension of the language model (e.g., 4096 for LLaMA-7B)

This projection is a translation layer. While the vision encoder speaks in visual features derived from pixel statistics and spatial patterns, the language model understands text embeddings representing discrete linguistic tokens. The projection layer learns to map visual concepts into linguistic representations that the frozen LLM can interpret. For example, the visual pattern of a "golden retriever" becomes embedded in a vector space location semantically close to the text token embeddings for "dog," "pet," and "canine."

Mathematically, the projection matrix $W$ learns to rotate and scale the vision feature space to align with the LLM's embedding subspace, such that dot products between projected visual features and text embeddings yield real similarity scores. This alignment lets the LLM to process visual information using the same attention mechanisms it uses for text, treating visual tokens as foreign words that have been translated into its native vocabulary. The visual tokens occupy real positions in the LLM's sequence, and the LLM's self-attention heads learn which visual tokens are needed to which text tokens when forming each output. No separate cross-modal attention module is required because the unified sequence format allows the language model's existing attention machinery to perform this integration naturally.

The final component is a pre-trained large language model, typically Vicuna (an instruction-tuned variant of LLaMA) or LLaMA itself. As established in earlier chapters on LLaMA architecture, these decoder-only transformers process sequences autoregressively to generate text, predicting each token based on all previous tokens in the sequence.

In LLaVA, the language model receives a hybrid sequence combining visual and textual information:

$$X = [H_v; H_t]$$

where:

• $X$: the complete input sequence to the language model
• $H_v$: the projected visual tokens (output from the projection layer)
• $H_t$: the embedded text tokens (word embeddings of the instruction and response)

The model processes this concatenated sequence through its transformer layers, applying causal masking to ensure autoregressive generation. The causal mask ensures that when predicting any given token, the model can attend to all visual tokens and all previously generated text tokens, but not to future text tokens. This maintains the left-to-right generation pattern needed for coherent text production. The visual tokens act as a "prefix" to the text, conditioning the generation on both image content and conversational history. Because the LLM processes the visual and textual tokens through the same self-attention mechanism, it can establish relationships between specific visual regions and linguistic concepts without requiring specialized cross-modal attention layers. A visual token representing a patch containing a "cat" can directly attend to the text token "feline" in the prompt, letting rich interplay between modalities within the standard transformer framework.

This design choice changes how the same transformer attention heads relate words to each other and words to image regions. The LLM does not need new machinery to handle visual input; it simply receives more tokens at the beginning of its context window. In practice, this means that capabilities the LLM has developed for reasoning about text, such as coreference resolution, logical inference, and step-by-step reasoning, naturally transfer to the multimodal setting because they operate on sequences of embeddings, and visual tokens are just another type of embedding.

The simplicity of this architecture is deliberate. By keeping both the vision encoder and language model frozen during initial training, LLaVA minimizes the number of trainable parameters to just the projection layer (and later, the LLM during fine-tuning). This parameter efficiency makes training feasible with modest compute budgets while using the full capabilities of large pre-trained models that would require hundreds of GPUs to train from scratch. The approach embodies the principle of compositional generalization: combining specialized, pre-trained modules to create capabilities greater than the sum of their parts.

The choice of connector architecture has generated substantial research interest. The original LLaVA paper used a single linear layer, in effect a matrix multiply with no bias or activation. This is the simplest possible connector, with only $d_v \times d_{\text{LLM}}$ parameters (about 4 million for CLIP ViT-L to LLaMA-7B). The fact that this minimal connector reaches real multimodal performance validates the structural compatibility hypothesis between CLIP and LLM embedding spaces.

More advanced connectors include the Q-Former used in BLIP-2, which employs a set of learned query tokens and a cross-attention mechanism to extract a fixed number of visual features from the full set of patch embeddings. This design can produce a compact, variable-length summary of image content rather than passing all 576 patch tokens to the LLM. The tradeoff is that Q-Former introduces significant additional complexity and training requirements, while the LLaVA linear projection remains extremely lean.

Subsequent ablation studies have shown that the two-layer MLP used in LLaVA 1.5 consistently outperforms both the linear layer and more complex connectors on standard benchmarks. The MLP is large enough to perform non-linear transformations that a linear layer cannot, but small enough to train quickly and reliably on modest data. This "Goldilocks" quality makes the MLP connector the default choice in most modern LLaVA-style systems.