LLaVA Architecture › Two-Stage Training Strategy

Training a multimodal model end-to-end from random initialization would require enormous datasets and computational resources, likely billions of image-text pairs and thousands of GPU hours. Such training would also risk destabilizing the carefully learned representations of pre-trained components. LLaVA's breakthrough insight is decomposing the training into two distinct stages, each with specific objectives and data requirements. This mirrors the progressive training approach discussed in earlier instruction tuning chapters, but extends it to the multimodal domain. The curriculum follows a pedagogical intuition: first teach the model to recognize what it sees, then teach it to converse about what it sees.

The first stage focuses exclusively on teaching the projection layer to translate visual features into language model embeddings. This stage is analogous to teaching a translator the basic vocabulary of a new language before asking them to interpret complex poetry. During this phase:

• The vision encoder remains frozen
• The language model remains frozen
• Only the projection matrix (or MLP) is trained

Training Data: This stage uses image-caption pairs from datasets like CC3M (Conceptual Captions), filtered to approximately 600,000 high-quality pairs. Unlike complex instruction data, these are simple descriptions: "a dog playing in the park" or "a red apple on a wooden table." These captions give direct, unambiguous mappings between visual content and linguistic descriptions. The simplicity is important: we want the projection layer to learn that a specific constellation of visual features corresponds to the concept "dog" before attempting to handle ambiguous or complex instructions.

The filtering step that produces 600K from the full 3.3M CC3M pairs is important. Raw web-scraped image captions often contain noise, off-topic text (like news article text where the image is incidental), or very short uninformative descriptions. Filtering by caption length, image quality, and relevance ensures the projection layer trains on clean signal. The selected pairs tend to have captions that accurately describe what is visually present, which is precisely the relationship the projection layer needs to learn.

Objective: The model learns to predict the caption text given the image. We minimize the cross-entropy loss between the predicted probability distribution over the vocabulary and the actual caption tokens. For a sequence of $L$ tokens, the loss is calculated by summing the negative log-likelihoods of each correct token:

$$\mathcal{L} = -\sum_{i=1}^{L} \log P(t_i \mid t_{<i}, H_v)$$

where:

• $\mathcal{L}$: the total cross-entropy loss across the caption
• $L$: the number of tokens in the target caption
• $t_i$: the $i$-th target token in the caption (the token we want to predict)
• $t_{<i}$: all tokens preceding position $i$ in the sequence (the context for prediction)
• $P(t_i \mid t_{<i}, H_v)$: the probability assigned by the model to the correct token $t_i$, conditioned on previous tokens $t_{<i}$ and projected visual features $H_v$
• $H_v$: the projected visual embeddings, serving as the visual context for the language model

This stage effectively aligns the visual and linguistic representations. The projection layer learns that certain visual patterns should map to specific semantic concepts in the language model's embedding space. Because the LLM is frozen, the projection layer must adapt to the LLM's existing semantic structure rather than vice versa. This constraint forces the projection to discover the "language" that the LLM already understands, effectively creating a visual vocabulary comprehensible to the text model.

During this stage, the projection layer is performing a form of reverse-engineering the LLM's embedding space. Given that the LLM knows what "dog" means, the projection layer must find the direction in $\mathbb{R}^{d_{\text{LLM}}}$ that activates the LLM's dog-related representations, and it must map CLIP's visual representation of a dog there. This is a well-defined optimization problem with a clear signal, and the 600K training pairs give enough variation (different dog breeds, lighting conditions, contexts) to learn a general mapping rather than an overfitted one.

Once the projection layer has learned basic vision-language alignment, the second stage teaches the model to follow complex, multimodal instructions. This stage adapts the instruction tuning paradigm to the visual domain. While Stage 1 teaches the model to describe, Stage 2 teaches it to converse and reason before following directions.

Modifications from Stage 1:

• The vision encoder remains frozen
• The projection layer continues training
• The language model is unfrozen and trained

Unfreezing the language model in Stage 2 is needed because caption prediction and instruction following require quite different behaviors. During Stage 1, the model only needs to output descriptions of image content. During Stage 2, it needs to answer questions, explain reasoning, refuse unreasonable requests, and adapt its response style to different instruction types. These higher-order behaviors require the language model's weights to update, incorporating visual context into the patterns that govern response generation.

Training Data: This stage requires instruction-following data with visual inputs. LLaVA pioneered the creation of such data using GPT-4V (or text-only GPT-4 with image captions) to generate diverse instruction-response pairs. The dataset includes:

• Conversational data: Multi-turn dialogues about images ("What color is the car? Is it moving?")
• Detailed description: Elaborate image captions with spatial relationships
• Complex reasoning: Visual question answering requiring inference ("Why might this person be carrying an umbrella?")

The data format follows standard chat templates:

Loss Masking: Following best practices from instruction tuning, the loss is computed only on the assistant's responses, not on the user's questions or the image tokens. This ensures the model learns to generate appropriate answers rather than simply memorizing the conversation structure. Without this masking, the model might learn to parrot back the question format or hallucinate image features, whereas masked training forces it to focus on creating relevant, grounded responses.

The two-stage approach creates a curriculum: first learn to see, then learn to converse about what you see. This separation prevents the instability that would arise from trying to learn both alignment and instruction following simultaneously. Attempting both at once would create conflicting gradients: the projection layer would receive mixed signals about whether to prioritize simple feature matching or complex reasoning, likely resulting in poor performance on both objectives.

The two-stage design has a remarkable compute advantage. Stage 1, which requires training only the projection layer, converges in roughly 4 hours on 8 A100 GPUs with the 600K CC3M subset. The projection layer has only about 16 million parameters for the ViT-L to LLaMA-7B setting, making each gradient update extremely cheap. By contrast, Stage 2 trains the full 7 billion parameter LLM, but uses only 150K instruction-following samples rather than billions of image-text pairs, keeping the total compute comparable to the fine-tuning budget for a text-only instruction tuned model.

This frugality is one of LLaVA's most important contributions. The original paper demonstrated that a single 8-GPU machine could complete the entire training pipeline in under a day, putting multimodal model training within reach of university research groups and small companies. Prior work required clusters of hundreds of GPUs and weeks of training. The democratizing effect of this efficiency has been significant: dozens of domain-specific and multilingual variants of LLaVA have been developed by groups that would not have had the resources to train a vision-language model from scratch.