Vision-Language Projection › Projection Training Strategies

Training the projection layer involves decisions about what components to freeze and how to structure the optimization process. These choices significantly impact computational cost, training stability, and final performance. The basic tension lies between preserving the capabilities of pre-trained models and adapting them to work together effectively.

Modern vision-language models typically employ one of three strategies:

Vision Frozen, LLM Frozen, Projection Trainable. This is the most common approach, exemplified by LLaVA and BLIP-2. The vision encoder and LLM retain their pre-trained weights, while only the projection parameters update. This preserves the capabilities of both pre-trained models while learning the interface between them. It requires the least compute and memory but limits the system's ability to adapt vision features to the specific demands of language understanding.

This freezing strategy rests on the assumption that both the vision encoder and language model have already learned general, transferable representations from their respective pre-training phases. By keeping them frozen, we prevent catastrophic forgetting, the phenomenon where a model loses previously learned capabilities when trained on new tasks. Since vision encoders like CLIP were trained on diverse image-text pairs and language models like LLaMA were trained on broad text corpora, freezing them maintains these general capabilities while the projection layer learns the specific translations needed for cross-modal interaction. The result is a system where the projection layer bears the entire burden of bridging the modalities, which can work well precisely because the pre-trained representations already have much of the semantic structure needed for the bridge to hold.

Vision Trainable, LLM Frozen. Sometimes used when the vision encoder is relatively small, or when domain adaptation is important. This allows visual representations to shift toward the language space but risks catastrophic forgetting of visual pre-training. If the vision encoder was pre-trained on natural images but the application involves medical imagery or satellite photos, unfreezing the vision encoder enables it to adapt its feature extraction to the new domain. The projection layer, now receiving domain-adapted visual features, can learn a cleaner mapping to the language space. However, this strategy requires careful regularization to prevent the vision encoder from drifting too far from its original representations, which could destabilize the projection.

Full Fine-tuning. Unfreezing all components often yields the best performance on specific tasks but requires massive compute and risks destabilizing either the vision or language representations. When the vision-language task involves specialized domains with distinct visual and linguistic patterns, such as scientific diagram understanding or technical document analysis, full fine-tuning allows the entire system to co-adapt. This approach demands significant GPU resources and careful tuning of learning rates across different components, as vision encoders and language models may require different optimization dynamics: vision encoders typically benefit from smaller learning rates to preserve their pre-trained visual features, while projection layers trained from scratch can tolerate larger learning rates to converge faster.

Many successful implementations use a two-stage approach to train projection layers effectively.

Stage 1 (Feature Alignment). Train the projection on large-scale image-text pairs with simple objectives, typically contrastive learning or image captioning. This stage establishes coarse alignment between modalities. The projection learns basic associations between visual concepts and their linguistic descriptions. For instance, visual features corresponding to furry animals, four legs, and pointed ears should map to linguistic embeddings near the word "dog." This alignment phase typically uses large, noisy datasets like LAION-400M or Conceptual Captions, where the abundance of data compensates for the simplicity of the training objective.

The first stage is often quick relative to later stages. Because the objective is simple and the model has no complex instruction-following behavior to learn, the projection can establish coarse alignment within a few hours on standard GPU hardware. This coarse alignment is sufficient to proceed to the second stage, where fine-grained adjustments take place.

Stage 2 (Instruction Tuning). Fine-tune on instruction-following data with the LLM's language modeling objective, adapting the projection for conversational and reasoning capabilities. As we discussed in the instruction tuning chapter, this second stage proves important for letting helpful assistant behaviors beyond simple captioning. The projection layer must learn to translate visual features into static descriptions and to present visual information in ways that support multi-turn reasoning, following complex instructions, and engaging in dialogue about image content.

The instruction tuning stage uses carefully curated datasets of image-question-answer triplets or multi-turn visual conversations. LLaVA used GPT-4 to generate such instruction-following data from image captions and bounding boxes, creating a dataset of around 150,000 examples that taught the model to follow instructions about visual content. During this stage, the projection learns which visual features correspond to which linguistic concepts and how to retrieve the relevant visual features given the context of a user's question.

When training with conversational data, it is needed to mask the loss on visual tokens and compute gradients only on text tokens. The projection layer receives gradients during text generation, learning to produce visual representations that facilitate accurate captioning or response generation, but we do not train the system to "reconstruct" images from text or to predict visual token values.

This masking strategy reflects the asymmetric nature of vision-language generation. During training, when the model processes a sequence like [visual tokens][text tokens: "A dog playing in the park"], the loss is computed only on predicting the text tokens "A dog playing in the park" given the visual tokens and any preceding text. The projection layer receives gradients flowing back from these text prediction losses, updating its weights to produce visual representations that better support accurate text generation, but no loss is applied to the visual tokens themselves.

Without loss masking, the model might be penalized for the raw numerical values of its visual token representations, which would create a misleading training signal: we want the visual tokens to be useful for generating text, not to take specific numerical values. The masking ensures that the only feedback the projection receives is whether the LLM can accurately generate the corresponding text given the visual representations, which is precisely the objective we care about.