LLaVA: Visual Instruction Tuning

Large Language and Vision Assistant (LLaVA) changes how we approach multimodal AI. Before LLaVA, the dominant approach for creating vision-language systems involved training large models from scratch on billion-scale image-text datasets, an approach that demanded extraordinary computational resources and often resulted in models that struggled to match the reasoning capabilities of pure language models. Rather than training large vision-language models from scratch, LLaVA shows that we can repurpose useful pre-trained components, a frozen vision encoder and a frozen large language model, connecting them through a surprisingly simple architectural bridge. This approach, known as visual instruction tuning, turns a text-only assistant into a multimodal conversationalist able to understand images, reason about visual content, and follow complex instructions that span both modalities.

The insight driving LLaVA is elegant in its efficiency. Computer vision has produced exceptional encoders like CLIP's Vision Transformer, which capture rich visual representations through extensive pre-training on hundreds of millions of image-text pairs. These encoders have learned to map visual content into rich, semantically real vector spaces where similar concepts cluster together. Separately, large language models like LLaMA and Vicuna have developed remarkable reasoning and instruction-following capabilities through extensive text-only training on trillions of tokens. These models have acquired world knowledge, logical reasoning abilities, and the capacity to follow complex instructions through next-token prediction. Rather than training a new model to acquire both skill sets simultaneously, which would require learning visual perception and linguistic reasoning from scratch, LLaVA asks a simpler question: can we teach these existing models to communicate with each other?

The answer is yes, through a lightweight projection layer that translates visual features into the linguistic embedding space that the language model understands. This chapter explores how LLaVA constructs this bridge, the two-stage training strategy that makes it possible, how visual instruction data is generated at scale, and how the architecture has evolved through subsequent iterations to handle higher resolutions and more complex visual reasoning tasks.

Before diving into the architecture, it is worth understanding why such a simple connection approach succeeds at all. The short answer is that both CLIP and large language models have already done the hardest work independently.

CLIP's training objective is explicitly cross-modal: it learns to align images and text descriptions in a shared embedding space using contrastive learning. An image of a cat and the text "a photograph of a cat" are pulled toward the same region of embedding space, while the image of a cat and the text "an elephant in water" are pushed apart. After training on 400 million image-text pairs, CLIP's visual features are not raw pixel statistics but semantic concepts, organized in a way that shows linguistic categories. The concept of "dog" in CLIP's visual space is already related to "canine," "pet," and "animal" in a way that mirrors how these words cluster in the embedding space of a language model.

This semantic overlap between CLIP's visual space and the language model's text space is what makes a simple linear projection viable. The projection layer does not need to teach the language model what a dog is. The LLM already knows. The projection layer only needs to translate the CLIP representation of a dog into the specific numerical coordinates the LLM uses internally for dog-related concepts. This is a much easier problem than building a bridge between entirely foreign spaces.

The language model contributes a complementary capability. Through instruction fine-tuning (covered in depth in earlier chapters), models like Vicuna have learned to follow instructions expressed as natural language. They understand "describe the image," "explain what is happening," and "what is the person on the left holding?" as requests requiring specific response patterns. What they lack is the perceptual input to ground these instructions in actual visual content. LLaVA supplies exactly that: a stream of semantically real visual tokens that the LLM can treat as additional context, much like a passage of text that describes the image.

The reason a linear projection suffices for initial alignment, and a two-layer MLP suffices for fine-grained alignment, is that the two representation spaces are already structurally compatible thanks to CLIP's contrastive training. This compatibility is the key empirical finding that LLaVA shows, and it has reshaped how the field thinks about multimodal architecture.

You can get a concrete sense of this compatibility by looking at nearest-neighbor relationships. Take any visual feature vector from CLIP's encoder for an image of, say, a sunset over mountains. Compute the cosine similarity between that vector and the word embedding vectors for hundreds of English words in a language model. You will find that "sunset," "horizon," "dusk," "mountains," and "orange" rank among the nearest neighbors, even though the CLIP encoder and the LLM were trained separately on different data with different objectives. This cross-model semantic coherence is not a coincidence. Both models were trained on internet data that shows the same underlying associations between visual and linguistic concepts in human culture.

The projection layer exploits this pre-existing coherence by learning a relatively simple transformation that aligns the scales and orientations of the two vector spaces. Researchers have verified this experimentally: after projecting CLIP features through a random (untrained) linear layer, the projected features still contain useful visual information. A trained projection layer simply optimizes this alignment more precisely, improving the LLM's ability to read the visual tokens accurately.

This also explains why LLaVA's approach generalizes across different LLM architectures. Whether you use LLaMA, Mistral, Phi, or Qwen as the language model backbone, the same CLIP vision encoder can be connected through a trained MLP and reach strong multimodal performance. The universal cross-architecture compatibility points to the underlying semantic structure being a property of internet-scale training data rather than any specific architectural choice.

LLaVA's architecture consists of three basic components working in concert. Understanding each piece and their interaction reveals why this design has become the template for modern open-source multimodal assistants. The elegance of this tripartite structure lies in its modularity: each component can be upgraded or modified independently, letting the architecture to evolve as better vision encoders or language models become available without requiring basic architectural changes.

LLaVA uses a pre-trained CLIP visual encoder, specifically the Vision Transformer (ViT) architecture. As discussed in prior chapters on Vision Transformers and CLIP, this encoder processes input images through patch embeddings and transformer layers to produce a sequence of visual feature vectors. The choice of CLIP is particularly strategic because CLIP has already learned to align visual features with linguistic concepts through its contrastive training objective. This means the vision encoder does not merely extract generic visual features; it produces representations that are already grounded in semantic meaning, making the subsequent alignment with the language model materially more tractable.

These visual tokens capture hierarchical visual information, from edges and textures in early layers to semantic concepts like objects and scenes in deeper layers. The transformer architecture processes these patches through self-attention mechanisms, letting each visual token to incorporate context from the entire image. This global receptive field means that even individual patch embeddings contain information about the full image content, not just their local region. A patch representing the corner of a room, for example, will have attended to the patches showing furniture and lighting as well as wall color, so its embedding implicitly encodes information about the room type and style.

LLaVA keeps this encoder frozen throughout training, preserving the rich visual representations learned from CLIP's extensive pre-training on image-text pairs. Freezing prevents catastrophic forgetting of the visual concepts CLIP has learned and maintains the stability of the visual feature distribution, which is needed for reliable multimodal alignment. If the vision encoder were allowed to drift during LLaVA training, the projection layer would be chasing a moving target, learning to project features from a shifting distribution rather than from a stable, well-understood encoder.

Another reason to freeze the vision encoder is parameter efficiency. The CLIP ViT-L/14 encoder contains roughly 307 million parameters. Allowing all of these to update during multimodal training would require materially more data and compute to avoid overfitting to the relatively small instruction tuning dataset. Freezing reduces the effective training problem to a much smaller parameter set, making convergence faster and more predictable.

The central innovation of LLaVA is the connector bridging vision and language modalities. Rather than employing complex cross-attention mechanisms, custom fusion layers, or detailed gating mechanisms that would require substantial architectural modifications to the language model, LLaVA uses a straightforward linear transformation (or multi-layer perceptron in later versions) to map visual features into the word embedding space of the language model. This simplicity is deliberate and powerful: it treats the alignment problem as a straightforward translation task between two fixed representation spaces.

Given vision features $Z \in \mathbb{R}^{N \times d_v}$, the projection layer produces language embeddings by learning a linear transformation from the vision feature space to the LLM's embedding space:

where:

• $H_v$: the projected visual embeddings in the LLM's input space (shape $N \times d_{\text{LLM}}$)
• $Z$: the visual feature matrix from the frozen vision encoder (shape $N \times d_v$)
• $W$: the trainable projection matrix mapping vision to language space (shape $d_v \times d_{\text{LLM}}$)
• $d_{\text{LLM}}$: the hidden dimension of the language model (e.g., 4096 for LLaMA-7B)

This projection is a translation layer. While the vision encoder speaks in visual features derived from pixel statistics and spatial patterns, the language model understands text embeddings representing discrete linguistic tokens. The projection layer learns to map visual concepts into linguistic representations that the frozen LLM can interpret. For example, the visual pattern of a "golden retriever" becomes embedded in a vector space location semantically close to the text token embeddings for "dog," "pet," and "canine."

Mathematically, the projection matrix $W$ learns to rotate and scale the vision feature space to align with the LLM's embedding subspace, such that dot products between projected visual features and text embeddings yield real similarity scores. This alignment lets the LLM to process visual information using the same attention mechanisms it uses for text, treating visual tokens as foreign words that have been translated into its native vocabulary. The visual tokens occupy real positions in the LLM's sequence, and the LLM's self-attention heads learn which visual tokens are needed to which text tokens when forming each output. No separate cross-modal attention module is required because the unified sequence format allows the language model's existing attention machinery to perform this integration naturally.

The final component is a pre-trained large language model, typically Vicuna (an instruction-tuned variant of LLaMA) or LLaMA itself. As established in earlier chapters on LLaMA architecture, these decoder-only transformers process sequences autoregressively to generate text, predicting each token based on all previous tokens in the sequence.

In LLaVA, the language model receives a hybrid sequence combining visual and textual information:

where:

• $X$: the complete input sequence to the language model
• $H_v$: the projected visual tokens (output from the projection layer)
• $H_t$: the embedded text tokens (word embeddings of the instruction and response)

The model processes this concatenated sequence through its transformer layers, applying causal masking to ensure autoregressive generation. The causal mask ensures that when predicting any given token, the model can attend to all visual tokens and all previously generated text tokens, but not to future text tokens. This maintains the left-to-right generation pattern needed for coherent text production. The visual tokens act as a "prefix" to the text, conditioning the generation on both image content and conversational history. Because the LLM processes the visual and textual tokens through the same self-attention mechanism, it can establish relationships between specific visual regions and linguistic concepts without requiring specialized cross-modal attention layers. A visual token representing a patch containing a "cat" can directly attend to the text token "feline" in the prompt, letting rich interplay between modalities within the standard transformer framework.

This design choice changes how the same transformer attention heads relate words to each other and words to image regions. The LLM does not need new machinery to handle visual input; it simply receives more tokens at the beginning of its context window. In practice, this means that capabilities the LLM has developed for reasoning about text, such as coreference resolution, logical inference, and step-by-step reasoning, naturally transfer to the multimodal setting because they operate on sequences of embeddings, and visual tokens are just another type of embedding.

The simplicity of this architecture is deliberate. By keeping both the vision encoder and language model frozen during initial training, LLaVA minimizes the number of trainable parameters to just the projection layer (and later, the LLM during fine-tuning). This parameter efficiency makes training feasible with modest compute budgets while using the full capabilities of large pre-trained models that would require hundreds of GPUs to train from scratch. The approach embodies the principle of compositional generalization: combining specialized, pre-trained modules to create capabilities greater than the sum of their parts.

The choice of connector architecture has generated substantial research interest. The original LLaVA paper used a single linear layer, in effect a matrix multiply with no bias or activation. This is the simplest possible connector, with only $d_v \times d_{\text{LLM}}$ parameters (about 4 million for CLIP ViT-L to LLaMA-7B). The fact that this minimal connector reaches real multimodal performance validates the structural compatibility hypothesis between CLIP and LLM embedding spaces.

More advanced connectors include the Q-Former used in BLIP-2, which employs a set of learned query tokens and a cross-attention mechanism to extract a fixed number of visual features from the full set of patch embeddings. This design can produce a compact, variable-length summary of image content rather than passing all 576 patch tokens to the LLM. The tradeoff is that Q-Former introduces significant additional complexity and training requirements, while the LLaVA linear projection remains extremely lean.

Subsequent ablation studies have shown that the two-layer MLP used in LLaVA 1.5 consistently outperforms both the linear layer and more complex connectors on standard benchmarks. The MLP is large enough to perform non-linear transformations that a linear layer cannot, but small enough to train quickly and reliably on modest data. This "Goldilocks" quality makes the MLP connector the default choice in most modern LLaVA-style systems.

Training a multimodal model end-to-end from random initialization would require enormous datasets and computational resources, likely billions of image-text pairs and thousands of GPU hours. Such training would also risk destabilizing the carefully learned representations of pre-trained components. LLaVA's breakthrough insight is decomposing the training into two distinct stages, each with specific objectives and data requirements. This mirrors the progressive training approach discussed in earlier instruction tuning chapters, but extends it to the multimodal domain. The curriculum follows a pedagogical intuition: first teach the model to recognize what it sees, then teach it to converse about what it sees.

The first stage focuses exclusively on teaching the projection layer to translate visual features into language model embeddings. This stage is analogous to teaching a translator the basic vocabulary of a new language before asking them to interpret complex poetry. During this phase:

• The vision encoder remains frozen
• The language model remains frozen
• Only the projection matrix (or MLP) is trained

Training Data: This stage uses image-caption pairs from datasets like CC3M (Conceptual Captions), filtered to approximately 600,000 high-quality pairs. Unlike complex instruction data, these are simple descriptions: "a dog playing in the park" or "a red apple on a wooden table." These captions give direct, unambiguous mappings between visual content and linguistic descriptions. The simplicity is important: we want the projection layer to learn that a specific constellation of visual features corresponds to the concept "dog" before attempting to handle ambiguous or complex instructions.

The filtering step that produces 600K from the full 3.3M CC3M pairs is important. Raw web-scraped image captions often contain noise, off-topic text (like news article text where the image is incidental), or very short uninformative descriptions. Filtering by caption length, image quality, and relevance ensures the projection layer trains on clean signal. The selected pairs tend to have captions that accurately describe what is visually present, which is precisely the relationship the projection layer needs to learn.

Objective: The model learns to predict the caption text given the image. We minimize the cross-entropy loss between the predicted probability distribution over the vocabulary and the actual caption tokens. For a sequence of $L$ tokens, the loss is calculated by summing the negative log-likelihoods of each correct token:

where:

• $\mathcal{L}$: the total cross-entropy loss across the caption
• $L$: the number of tokens in the target caption
• $t_i$: the $i$-th target token in the caption (the token we want to predict)
• $t_{<i}$: all tokens preceding position $i$ in the sequence (the context for prediction)
• $P(t_i \mid t_{<i}, H_v)$: the probability assigned by the model to the correct token $t_i$, conditioned on previous tokens $t_{<i}$ and projected visual features $H_v$
• $H_v$: the projected visual embeddings, serving as the visual context for the language model

This stage effectively aligns the visual and linguistic representations. The projection layer learns that certain visual patterns should map to specific semantic concepts in the language model's embedding space. Because the LLM is frozen, the projection layer must adapt to the LLM's existing semantic structure rather than vice versa. This constraint forces the projection to discover the "language" that the LLM already understands, effectively creating a visual vocabulary comprehensible to the text model.

During this stage, the projection layer is performing a form of reverse-engineering the LLM's embedding space. Given that the LLM knows what "dog" means, the projection layer must find the direction in $\mathbb{R}^{d_{\text{LLM}}}$ that activates the LLM's dog-related representations, and it must map CLIP's visual representation of a dog there. This is a well-defined optimization problem with a clear signal, and the 600K training pairs give enough variation (different dog breeds, lighting conditions, contexts) to learn a general mapping rather than an overfitted one.

Once the projection layer has learned basic vision-language alignment, the second stage teaches the model to follow complex, multimodal instructions. This stage adapts the instruction tuning paradigm to the visual domain. While Stage 1 teaches the model to describe, Stage 2 teaches it to converse and reason before following directions.

Modifications from Stage 1:

• The vision encoder remains frozen
• The projection layer continues training
• The language model is unfrozen and trained

Unfreezing the language model in Stage 2 is needed because caption prediction and instruction following require quite different behaviors. During Stage 1, the model only needs to output descriptions of image content. During Stage 2, it needs to answer questions, explain reasoning, refuse unreasonable requests, and adapt its response style to different instruction types. These higher-order behaviors require the language model's weights to update, incorporating visual context into the patterns that govern response generation.

Training Data: This stage requires instruction-following data with visual inputs. LLaVA pioneered the creation of such data using GPT-4V (or text-only GPT-4 with image captions) to generate diverse instruction-response pairs. The dataset includes:

• Conversational data: Multi-turn dialogues about images ("What color is the car? Is it moving?")
• Detailed description: Elaborate image captions with spatial relationships
• Complex reasoning: Visual question answering requiring inference ("Why might this person be carrying an umbrella?")

The data format follows standard chat templates:

Loss Masking: Following best practices from instruction tuning, the loss is computed only on the assistant's responses, not on the user's questions or the image tokens. This ensures the model learns to generate appropriate answers rather than simply memorizing the conversation structure. Without this masking, the model might learn to parrot back the question format or hallucinate image features, whereas masked training forces it to focus on creating relevant, grounded responses.

The two-stage approach creates a curriculum: first learn to see, then learn to converse about what you see. This separation prevents the instability that would arise from trying to learn both alignment and instruction following simultaneously. Attempting both at once would create conflicting gradients: the projection layer would receive mixed signals about whether to prioritize simple feature matching or complex reasoning, likely resulting in poor performance on both objectives.

The two-stage design has a remarkable compute advantage. Stage 1, which requires training only the projection layer, converges in roughly 4 hours on 8 A100 GPUs with the 600K CC3M subset. The projection layer has only about 16 million parameters for the ViT-L to LLaMA-7B setting, making each gradient update extremely cheap. By contrast, Stage 2 trains the full 7 billion parameter LLM, but uses only 150K instruction-following samples rather than billions of image-text pairs, keeping the total compute comparable to the fine-tuning budget for a text-only instruction tuned model.

This frugality is one of LLaVA's most important contributions. The original paper demonstrated that a single 8-GPU machine could complete the entire training pipeline in under a day, putting multimodal model training within reach of university research groups and small companies. Prior work required clusters of hundreds of GPUs and weeks of training. The democratizing effect of this efficiency has been significant: dozens of domain-specific and multilingual variants of LLaVA have been developed by groups that would not have had the resources to train a vision-language model from scratch.

A key innovation enabling LLaVA's capabilities is the methodology for creating visual instruction-following data at scale. Manually annotating millions of multimodal conversations would be prohibitively expensive, requiring thousands of human hours to craft diverse, high-quality examples covering the full spectrum of visual reasoning tasks. Since manual annotation is infeasible at the scale required for training modern multimodal models, LLaVA uses existing language models to bootstrap the dataset, using the strong zero-shot capabilities of GPT-4V to generate training supervision.

The original LLaVA paper used GPT-4V (or GPT-4 with caption and bounding boxes) to generate training data. This bootstrapping approach exploits the fact that while GPT-4V is expensive to use at inference time for all users, it can be used cost-effectively to generate high-quality training data that teaches smaller, more efficient models to replicate its capabilities. The process works as follows:

1. Image Captioning: Generate detailed captions describing image contents, including objects and attributes as well as activities
2. Bounding Box Annotation: Identify objects and their locations (when using text-only models, bounding boxes give spatial grounding)
3. Prompt Engineering: Feed captions and bounding boxes to GPT-4 with prompts requesting specific types of interactions

For example, to generate conversational data:

This produces natural, diverse dialogues that capture the distribution of real user queries. The approach generates three types of instruction data:

• Conversation (58K samples): Multi-turn QA about image content, simulating realistic user-assistant interactions
• Detailed Description (23K samples): Long-form image descriptions that capture detailed visual details beyond simple captioning
• Complex Reasoning (77K samples): Questions requiring inference and external knowledge, testing the model's ability to reason about causality and intent alongside abstract concepts

The diversity of these categories ensures that the trained model can handle simple recognition tasks, open-ended dialogue, and analytical reasoning about visual content.

The synthetic data approach allows for systematic coverage of visual reasoning skills across multiple dimensions of understanding:

• Object recognition: Identifying items in the scene, from common objects to rare or specialized items
• Attribute detection: Colors and sizes, materials and textures, plus other physical properties
• Spatial relationships: Positional reasoning ("left of," "behind," "above") and compositional scene understanding
• Action understanding: What people or animals are doing, including subtle activities and interactions
• Abstract reasoning: Intentions, emotions, likely future events, and causal relationships

The complex reasoning category deserves particular attention because it is the hardest to obtain through human annotation. Questions like "Why might this storefront have bars on its windows?" or "What time of day does this photo most likely stand for?" require drawing on world knowledge beyond what is literally visible. GPT-4V's strong common-sense reasoning makes it well-suited to generate such examples, and these samples are precisely what teach LLaVA to go beyond object detection toward real visual understanding.

However, synthetic data has limitations that must be carefully managed. GPT-4V may hallucinate details not present in the image, generating confident descriptions of objects that do not exist, or generate questions that assume knowledge the image does not give (such as asking about a person's name when only their back is visible). The original LLaVA paper included examples where the generated questions and answers were inconsistent with the provided image metadata, illustrating that this noise is real even with a powerful generator.

Later versions of LLaVA addressed this by incorporating human-verified academic datasets and more advanced filtering mechanisms to remove low-quality or hallucinated examples from the training mix. The key insight from this evolution is that the quality distribution of synthetic data matters more than its raw volume: a smaller, carefully curated synthetic dataset outperforms a much larger but noisy one.

One subtle aspect of the original data generation pipeline is that text-only GPT-4 was used in some cases (without direct image access), relying on image captions and bounding box coordinates as proxies for visual content. This is a clever workaround for cases where GPT-4V access is limited or expensive: since COCO images already have detailed bounding box annotations, one can describe the spatial layout of an image entirely in text and ask GPT-4 to generate conversational data about it.

For example, given a bounding box annotation saying "person at (120, 80, 200, 300), umbrella at (100, 40, 220, 120)," GPT-4 can generate the question "The person in the image is carrying an umbrella. What might this suggest about the weather?" without ever seeing the actual image. This approach lets data generation at scale using text-only models, which were far cheaper and more accessible than GPT-4V when LLaVA was originally developed.

The limitation is that bounding box proxies cannot capture aesthetic qualities, lighting, emotional tone, or fine visual details that a human or GPT-4V would notice. These proxies work well for object-centric questions but struggle for questions about overall mood, artistic style, or subtle interactions. This limitation motivated LLaVA 1.5 to use richer image descriptions and incorporate GPT-4V directly for higher-quality data generation.

LLaVA 1.5 introduced several necessary improvements that materially boosted performance without fundamentally changing the architecture. These refinements show how careful engineering choices in data and projection design matter as much as the core concept. By iterating on the connector design and training data mixture, the authors showed that the simple LLaVA framework had substantial headroom for improvement without requiring architectural complexity.

The original LLaVA used a simple linear projection from vision to language space. While effective, this linear mapping assumes a relatively simple relationship between visual and linguistic representations. LLaVA 1.5 replaces this with a two-layer MLP with GELU activation. This non-linear mapping transforms visual features through an intermediate representation before projecting to the LLM space:

where:

• $Z$: the visual feature vectors from the frozen vision encoder (shape $N \times d_v$)
• $W_1, b_1$: the weight matrix and bias for the first linear layer (mapping from $d_v$ to $d_{\text{LLM}}$)
• $\text{GELU}$: the Gaussian Error Linear Unit activation function, introducing non-linearity between layers
• $H_{\text{mid}}$: the intermediate representation after the first linear layer and activation
• $W_2, b_2$: the weight matrix and bias for the second linear layer (mapping within $d_{\text{LLM}}$)
• $H_v$: the final projected visual embeddings ready for input to the LLM

Why does the MLP outperform the linear projection? Consider what the linear layer can and cannot stand for. A linear map can rotate and scale while shearing the visual feature space, but it cannot introduce curvature or stand for non-linear relationships between visual concepts. For instance, if "red" and "round" are both present in the visual features, a linear layer can adjust their magnitudes but cannot compute "red AND round implies apple" in any way that departs from a linear combination. The intermediate hidden layer and GELU activation allow the network to compose visual features in non-linear ways, capturing richer semantic structure before projecting to the LLM space.

This non-linear connector gives greater capacity to map complex visual concepts to linguistic embeddings. The added parameters (though still small relative to the LLM, typically less than 1% of total model parameters) allow for more fine-grained translation of visual features, particularly for abstract concepts and fine-grained visual details that might not align linearly with text embeddings.

LLaVA 1.5 incorporates additional training data from academic visual question answering datasets (VQA-v2, GQA, OKVQA, OCR-VQA) during the second stage. These datasets give high-quality, human-annotated question-answer pairs covering diverse reasoning types including compositional questions, reading comprehension, and knowledge-based reasoning.

However, rather than using the datasets' original formats, which often include dataset-specific biases or formatting artifacts, LLaVA 1.5 reformats them into simple, consistent prompts. Instead of complex VQA templates like "What is the color of the object left of the person?", the data uses direct questions: "What color is the car?" This approach maintains the diversity of academic datasets while preventing the model from overfitting to specific dataset formats or answer distributions. It also shows that visual instruction tuning can subsume traditional visual QA benchmarks when properly formatted. By stripping away dataset-specific formatting, the model learns the underlying reasoning capabilities rather than memorizing stylistic patterns associated with particular benchmarks.

The combination of synthetic GPT-generated data and curated academic benchmarks creates a training mixture that is both diverse (from the synthetic data) and reliable (from human-verified academic annotations). This dual-source strategy has become standard practice in modern multimodal training pipelines, and the balance between synthetic and human-verified data remains an active research question.

Original LLaVA used 224px or 336px image resolution, which limits the model's ability to perceive fine details such as small text, distant faces, or detailed textures. LLaVA 1.5 supports higher resolutions by dividing images into grids and processing each crop separately through the vision encoder.

The tiling approach works by dividing a high-resolution image into non-overlapping crops, each of size matching the vision encoder's native input resolution (336px). A 672px image yields a 2x2 grid of four 336px crops, each processed independently. The visual tokens from all crops are concatenated and projected together, letting the model to examine image regions at higher fidelity without requiring a modified vision encoder. For example, if a document image contains small text that falls within a single 336px crop, the model can read it by examining that crop's tokens at full resolution, even though the full document would be too large for the encoder to process at once.

This tiling approach is not without cost. A 2x2 grid quadruples the number of visual tokens from 576 to 2304, a 3x3 grid produces 5184 tokens, and each token adds to the sequence length that the LLM must process through its quadratic attention mechanism. Longer sequences mean quadratically increasing memory and computation at inference time, creating a direct tradeoff between visual resolution and processing cost that scales unfavorably for very high-resolution images.

Building on the grid approach of 1.5, LLaVA-NeXT (also called LLaVA 1.6) introduces dynamic high-resolution processing and improved visual reasoning capabilities. This iteration addresses one of the basic limitations of fixed-resolution vision encoders: the inability to adapt to varying image sizes and aspect ratios while maintaining computational efficiency.

Rather than resizing all images to a fixed square, which distorts aspect ratios and can obscure important details in non-square images, LLaVA-NeXT maintains aspect ratios and processes images at their native resolution (up to a maximum). The strategy involves several key mechanisms:

Aspect Ratio Preservation: Images are divided into grids that match predefined aspect ratio buckets (1:1, 1:2, 2:1, 1:3, 3:1, and so on). Rather than stretching a wide panoramic image into a square, the system selects the grid setup whose total shape most closely matches the image's native aspect ratio, then pads only minimally to fill the chosen grid. This bucketing system ensures that images are processed without the geometric distortion caused by naive resizing to squares. A person photographed in portrait orientation is processed in a tall grid, not a distorted square.

Variable Token Length: Different images produce different numbers of visual tokens based on their resolution and aspect ratio. A panoramic image might generate twice as many tokens as a square image of similar total area, letting the model to process wide scenes without losing detail on either the left or right extremes. This variability requires the LLM to handle sequences of differing lengths, which modern transformer architectures accommodate through dynamic attention masking.

Global Context Thumbnail: Alongside the tiled high-resolution crops, LLaVA-NeXT always includes a downsampled version of the full image as a global context thumbnail, processed through the same vision encoder at standard resolution. This global thumbnail gives the model an overall view of the scene even when the tiled crops are individually small excerpts. Without this thumbnail, the model examining individual tiles might lose track of the overall spatial composition, mistaking a foreground object tile for the entire scene. The thumbnail ensures that the model can answer questions like "what is in the background of this image" even when the high-resolution processing partitions the image into small patches.

This approach allows the model to read small text in images, recognize fine details in complex scenes, and handle documents and charts more effectively than fixed-resolution approaches. The combination of global context and local detail is particularly powerful for document understanding tasks where a user might ask "summarize this page" (requiring global understanding) and "what does the footnote say" (requiring local high-resolution detail) in the same conversation.

LLaVA-NeXT employs more advanced data mixing strategies, balancing multiple types of visual understanding:

• General visual conversation for everyday scene understanding
• OCR and document understanding for text-rich images
• Multi-language visual data to support non-English queries
• Scientific diagrams and charts for specialized domain knowledge

The training recipe also extends to different base LLMs, including Mistral and other modern architectures beyond the original Vicuna. This shows the generalizability of the LLaVA approach. By varying the base language model, researchers can confirm that the visual instruction tuning methodology is reliable across diverse underlying language architectures, and that the key capability resides in the training pipeline rather than in any particular LLM's architecture.

Let us examine how LLaVA functions in practice, from processing an image through generating a response. We'll implement a simplified version of the forward pass to understand the data flow.

The key architectural parameters for LLaVA determine the model's capacity, resolution capabilities, and computational requirements. Understanding these parameters helps calibrate expectations when adapting LLaVA for different applications or when choosing between model scales.

• vision_hidden_size: Dimension of the vision encoder's output features (1024 for CLIP ViT-L/14). This determines the size of visual representations before projection. Higher dimensions capture more visual detail but require larger projection matrices. Smaller CLIP variants use 768-dimensional features, while larger models can use up to 1280 dimensions.
• llm_hidden_size: Dimension of the language model's embedding space (4096 for LLaMA-7B). The projection layer maps vision features to this dimension. This is typically the largest dimension in the system and determines the width of all subsequent transformer layers. LLaMA-13B uses 5120 dimensions, and larger models scale further.
• num_patches: Number of visual tokens extracted from the image (576 for 336px resolution with 14px patches). This controls the spatial resolution of visual input. The formula is $(H / P) \times (W / P)$ where $H$ and $W$ are the image height and width and $P$ is the patch size. Higher resolution images processed with tiling can generate thousands of tokens.
• MLP connector: A two-layer projection network with GELU activation that transforms vision features to language space. The intermediate dimension typically matches $d_{\text{LLM}}$ (4096), maintaining constant width through the projection and giving the network enough capacity for non-linear alignment.

The number of visual tokens per image has a particularly strong influence on inference latency. With 576 tokens for a single 336px image, adding an image to the context is equivalent to adding a paragraph of text in terms of sequence length. For a 2x2 tiled image at 672px resolution, this becomes 2304 tokens, comparable to a full page of text. This has direct implications for user-facing applications where inference speed matters: models serving high-resolution image queries face a materially higher compute burden than those handling text-only queries or lower-resolution images.

Understanding how LLaVA trains requires examining the specific loss computation, particularly the handling of multimodal sequences and label masking. The training process must carefully balance supervision across different parts of the input to prevent the model from learning spurious correlations or trivial solutions.

A training example in LLaVA consists of a sequence combining visual markers, visual features, and text tokens:

where:

• $S$: the complete multimodal input sequence
• $\text{<image>}$: a special token marking the image position (often replaced by projected visual features)
• $V_1, \ldots, V_N$: the projected visual tokens representing image patches (typically 576 tokens for 336px resolution)
• $\text{<text\_tokens>}$: the sequence of tokenized instruction and response text

The full input to the LLM combines the projected visual features with embedded text tokens:

where:

• $\text{Projected}(Z)$: the visual features after transformation through the projection layer
• $T_{\text{instruction}}$: the token IDs representing the user's question or prompt text
• $T_{\text{response}}$: the token IDs representing the assistant's target answer
• $\text{Embed}(\cdot)$: the language model's embedding lookup function converting token IDs to dense vectors
• $\text{Inputs}$: the final concatenated sequence of embeddings fed to the transformer layers

This concatenation treats visual tokens and text tokens identically once they enter the transformer, letting the self-attention mechanism to establish relationships between any visual patch and any text token without architectural constraints on cross-modal interaction. There is no separate "image processing" module inside the transformer: visual tokens and text tokens flow through the same attention heads, the same feed-forward networks, and the same layer normalization operations.

As discussed in earlier chapters on instruction tuning, we compute loss only on the tokens we want the model to learn to generate. For LLaVA:

1. Image tokens: No loss computed (the model does not predict these)
2. User instruction tokens: No loss computed (we do not train the model to predict the question)
3. Assistant response tokens: Cross-entropy loss computed
4. Padding tokens: Ignored in loss computation

This masking ensures the model learns to generate appropriate responses conditioned on images and questions, rather than memorizing the questions themselves or trying to reconstruct image features. Training on user questions would encourage the model to parrot back query patterns rather than learn to answer them, while training on image tokens would waste capacity on reconstructing visual inputs that are already provided directly.

The loss at each unmasked position is:

where $\theta$ is all trainable parameters. The total loss averages over unmasked positions:

where $\mathcal{A}$ denotes the set of active (non-masked) token positions. Averaging over active tokens rather than all positions ensures that examples with short responses receive the same effective learning rate per response token as examples with long responses, preventing the model from implicitly down-weighting short answers.

During Stage 2, gradients flow through the entire system except the frozen vision encoder. Starting from the cross-entropy loss on assistant response tokens, gradients propagate backward through the LLM's transformer layers, through the projection layer, and terminate at the projection layer's inputs (which are the frozen vision encoder outputs). This means the vision encoder's parameters never receive updates, but the projection layer learns from the signal that originates in the LLM's loss computation.

This gradient flow has an important implication for what the projection layer learns to stand for. In Stage 1, the projection learns to map visual features to language space in a way that produces correct captions, a relatively local objective. In Stage 2, the projection layer receives gradients that reflect whether its output contributed to correct instruction-following behavior, a much richer and more demanding objective. The Stage 2 gradient signal teaches the projection to produce semantically accurate visual tokens that are useful for the LLM's reasoning processes.

The learning rates used in each stage reflect this difference in objectives. Stage 1 typically uses a higher learning rate (around 1e-3) since the projection layer starts from random initialization and must learn rapidly from a relatively simple objective. Stage 2 uses a much lower learning rate for both the projection layer (around 2e-5) and the LLM (1e-4 to 2e-5 depending on scale), as both are making incremental refinements rather than large initial adjustments.

Before discussing LLaVA's limitations, it is useful to understand how multimodal models are evaluated, because evaluation criteria shape what we consider a "good" or "bad" model and what failures are most consequential in practice.

Multimodal evaluation draws from several families of benchmarks, each testing different aspects of visual understanding:

Visual Question Answering (VQA): Datasets like VQA-v2 and GQA present images with multiple-choice or open-ended questions. VQA-v2 was designed to reduce language bias so that questions have balanced answer distributions (roughly half yes, half no for yes/no questions), forcing models to use visual information rather than relying on linguistic priors. GQA focuses on compositional reasoning, with questions that require combining multiple visual facts (for example, "Is the item to the left of the bench a chair?").

ScienceQA tests multimodal scientific reasoning, presenting diagrams and charts alongside domain-specific images and questions that require integrating visual and textual knowledge. This benchmark is particularly demanding because the visual content is often abstract (diagrams rather than photographs) and the questions require understanding scientific concepts, not just object recognition.

MMBench and MME are broader evaluation suites designed to assess a wide range of visual capabilities including object existence, counting, position, color recognition, OCR, commonsense reasoning in a standardized format. These complete suites were developed because narrow benchmarks like VQA-v2 could be "solved" by models that were weak on other visual tasks.

TextVQA and OCR-VQA specifically test text recognition and reasoning over images containing text. These benchmarks are particularly sensitive to resolution: at 224px, small text is often illegible to the vision encoder, and model performance on these benchmarks improved substantially when LLaVA 1.5 introduced higher-resolution processing.

Numerical benchmarks capture some aspects of visual understanding but miss others, particularly conversational quality, instruction following, and the ability to handle open-ended queries that do not have a single correct answer. The original LLaVA paper introduced LLaVA-Bench, a set of 60 diverse in-the-wild images paired with 3 types of instructions: conversational, detailed description, and complex reasoning. Responses are evaluated by GPT-4V on a scale of 1-10, creating a relative score that compares the evaluated model to a reference model.

This LLM-as-judge evaluation paradigm, where a powerful model scores free-form responses, has become increasingly popular in the field because it captures aspects of response quality that automatic metrics (like BLEU or CIDEr) miss: factual accuracy and response depth as well as coherence and relevance to the specific visual content. However, this approach has its own biases, including a tendency for the judge model to prefer longer responses regardless of their accuracy and to favor outputs that stylistically resemble the judge's own generation patterns.

Object hallucination has received its own dedicated evaluation methodology. The POPE (Polling-based Object Probing Evaluation) benchmark asks adversarial yes/no questions about whether specific objects are present in an image, using objects that are absent but semantically plausible given the scene context. A model with strong visual grounding will answer "no" correctly; a model that hallucinates will confirm the presence of objects that were never in the image.

LLaVA models show moderate hallucination rates on POPE, performing better than early baselines but lagging behind models specifically trained with anti-hallucination objectives. This is an active area of improvement, and later models in the LLaVA lineage incorporate dedicated training steps to reduce hallucination, including reinforcement learning from human feedback and contrastive visual grounding.

While LLaVA's architectural simplicity lets rapid experimentation and deployment, it also imposes constraints that subsequent research has sought to address. Understanding these limitations is needed for determining when LLaVA-style architectures are appropriate and when more complex approaches might be necessary.

The original LLaVA architecture processes images at fixed resolutions (336px), limiting its ability to perceive fine details like small text, distant objects, or subtle textures. While LLaVA-NeXT addresses this through dynamic tiling, the approach remains computationally expensive as each tile generates hundreds of tokens. Processing a high-resolution image might require 2,000 to 4,000 visual tokens, compared to 576 for standard resolution, increasing memory usage and computation quadratically with sequence length due to the self-attention mechanism.

This creates a basic tension between visual fidelity and computational efficiency that differs from human perception. Human vision uses a fovea that selectively processes detail only in the region of current attention, leaving peripheral vision at low resolution. A viewer reading a sign in an image focuses on the sign region without wasting perceptual resources on the background. LLaVA's tiling approach processes all tiles equally, with no mechanism to allocate more computation to regions that the user's query asks about. Building selective visual attention based on textual queries remains an open research problem.

LLaVA's architecture maintains a strict separation between vision and language processing. The vision encoder operates independently, creating a fixed set of tokens regardless of the textual query. This means the visual processing is query-agnostic: the same visual tokens are generated whether the user asks "What is in the top left corner?" or "Describe the overall mood of this scene."

Unlike models with true cross-attention between modalities, LLaVA cannot dynamically focus visual processing based on the question being asked. If you ask "What color is the small sign in the background?" the model processes the entire image with equal fidelity, unable to direct the vision encoder to attend specifically to that region. The LLM must do all the visual focusing work through attention over the fixed pool of visual tokens, which requires the needed patch token to have adequately captured the sign's features even without the benefit of query-directed attention.

This limitation motivated architectures like Flamingo, which we'll explore in the next chapter, that employ perceiver resamplers and gated cross-attention to let dynamic interaction between modalities. These mechanisms allow the language model to selectively query visual information needed to the current prompt, rather than receiving a static visual summary.

One of the most studied failure modes of LLaVA-style models is object hallucination: confidently describing objects, attributes, or relationships that are not present in the image. This occurs because the language model's strong priors about what "usually" appears in similar contexts can override the actual visual evidence. A model that has seen thousands of kitchen images during training might describe a knife even in a kitchen photo where no knife is visible, because the LLM's textual prior says "kitchens often contain knives."

This failure mode is partly architectural. The visual tokens give an average spatial representation of the image, but the LLM's cross-token attention can be dominated by strong textual patterns if the visual signal for a particular region is weak or ambiguous. Mitigation strategies include reinforcement learning from human feedback specifically targeting hallucination, contrastive training that penalizes descriptions inconsistent with the image, and decoding strategies that periodically re-check generated claims against the visual evidence.

LLaVA inherits biases from both its components. The CLIP encoder may struggle with uncommon visual concepts not well-represented in its training data, such as rare species, specialized equipment, or cultural artifacts from underrepresented regions. Simultaneously, the language model may generate confident but incorrect responses based on textual biases or outdated knowledge. The synthetic data generation process, while scalable, can produce questions that assume Western cultural contexts or over-stand for objects common in internet images, potentially limiting the model's performance on culturally diverse content.

The CC3M and COCO datasets used for training are heavily biased toward English-language images from Western contexts. Images of food and clothing alongside architecture and social settings from other cultures are materially underrepresented. Models trained on such data show lower performance on images from under-represented cultural contexts, a problem that the multilingual extensions introduced in LLaVA-NeXT begin to address but have not fully solved.

Despite these limitations, LLaVA's impact on multimodal AI cannot be overstated. It established the paradigm that powers most open-source multimodal assistants today. By showing that a simple projection layer suffices to align pre-trained vision and language models, LLaVA democratized multimodal AI development, letting researchers with modest compute resources (single GPUs or small clusters) to create competitive vision-language models. Prior to LLaVA, multimodal research required large infrastructure; after LLaVA, individual researchers could fine-tune multimodal assistants for specific domains or languages.

The architectural pattern of frozen vision encoder, lightweight connector, and frozen or lightly fine-tuned LLM has become the standard approach for efficiently adding vision capabilities to language models. This design philosophy extends beyond vision to other modalities, inspiring similar approaches for audio (connecting audio encoders like Whisper to language models), video (connecting frame-level visual encoders to language models with temporal pooling), and structured data integration. The principle of modality alignment through lightweight projection layers has proven remarkably general, suggesting that the key to multimodal AI lies not in training monolithic models from scratch, but in learning to translate between the rich representational spaces of specialized pre-trained models.

LLaVA also demonstrated the power of GPT-4V-assisted data generation as a training paradigm. The idea of using a powerful but expensive model to generate training data for a cheaper, deployable student model has since been applied across many areas of NLP and multimodal AI. This teacher-student pipeline, where GPT-4V acts as the teacher and LLaVA as the student, enabled rapid capability transfer without requiring direct access to GPT-4V for every user interaction.

The open-source release of LLaVA weights and training code further amplified its impact. Within months of the initial publication, research groups had adapted LLaVA for medical imaging, satellite image analysis, multilingual visual question answering, and robotics perception, domains where GPT-4V's API costs and data privacy concerns made a proprietary model impractical. This ecosystem of specialized variants validated the modularity of the LLaVA design: swapping in domain-specific vision encoders or language models proved straightforward, confirming that the architectural principle generalizes beyond the specific components used in the original paper.

LLaVA shows that multimodal intelligence emerges not from training large models from scratch, but from teaching existing expert models to communicate. The architecture's elegance lies in its restraint: by freezing both the vision encoder and language model during initial training, LLaVA focuses learning entirely on the translation layer between modalities, achieving remarkable capabilities with minimal trainable parameters.

Key insights from the LLaVA approach include:

• Modular alignment: Pre-trained vision encoders and language models can be connected through lightweight trainable projections, preserving the capabilities of both while letting cross-modal reasoning. This compositional approach allows each component to be updated independently as better models become available.

• Curriculum training: Separating feature alignment (Stage 1) from instruction tuning (Stage 2) creates stable training dynamics and prevents catastrophic forgetting of pre-trained capabilities. This staged approach mirrors human learning, where basic perception develops before complex linguistic reasoning.

• Visual instruction tuning: Extending instruction tuning to the multimodal domain requires carefully masked loss computation and diverse instruction data, which can be effectively generated using existing language models. This data-centric approach lets rapid scaling of training sets without large human annotation efforts.

• Architectural evolution: Simple changes, upgrading from linear to MLP projections and supporting dynamic resolutions, yield large improvements without abandoning the core approach. This suggests that the basic insight of modality alignment via projection is reliable and extensible.

• Emergent multimodal reasoning: Because visual tokens are processed by the same transformer attention mechanism as text tokens, the LLM's existing reasoning capabilities (logical inference, multi-step problem solving, commonsense reasoning) naturally extend to visual inputs without any specialized training on reasoning with images.

As we progress to the Flamingo architecture in the next chapter, we'll examine alternative approaches that sacrifice some of LLaVA's simplicity for greater expressiveness in cross-modal interactions, introducing mechanisms for dynamic attention between vision and language that LLaVA's frozen encoder design deliberately avoids.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about LLaVA architecture, training strategies, and multimodal alignment.