Vision Encoders for VLMs › Training and Fine-tuning Dynamics

Understanding how vision encoders are trained and adapted for VLMs is needed for making good architectural choices. The interplay between the encoder, the projection layer, and the language model during training determines how well the final system generalizes across visual tasks.

Most production VLMs train their vision components in multiple stages, each serving a distinct purpose:

Stage 1: Vision encoder pretraining. The vision encoder is trained independently on a large image-text dataset using a contrastive objective such as SigLIP or InfoNCE. At this stage, the language backbone is not involved. The encoder learns to produce image embeddings that align with text embeddings from a separate text encoder. The scale of this pretraining is enormous: SigLIP-SO400M was trained on roughly 4 billion image-text pairs, giving it broad visual coverage before any multimodal instruction tuning occurs.

Stage 2: Cross-modal projection alignment. A lightweight projection layer, typically a two-layer MLP or a linear projection, is trained to map vision encoder outputs into the language model's token embedding space. During this stage, both the vision encoder and language backbone are usually frozen, and only the projection layer is updated. The projection layer learns to translate the statistical properties of vision embeddings into a distribution that the language model can interpret as meaningful token-like inputs. This stage is computationally cheap relative to the full training, requiring only a few hundred million to a few billion image-text pairs.

Stage 3: Instruction tuning. The full system is fine-tuned on instruction-following data that pairs images with questions and answers. At this stage, the language model is updated to learn how to use visual inputs to generate helpful responses. The vision encoder may remain frozen, or a small subset of its final layers may be unfrozen to allow limited adaptation. This stage uses far less data than pretraining, typically tens of millions of examples, but requires high-quality annotations.

This three-stage approach reflects a principled decomposition of the learning problem. Pretraining the encoder separately ensures it develops reliable visual representations before the noise introduced by language model gradients can interfere. Projection alignment ensures the modalities can communicate before the full model is trained to follow instructions. Instruction tuning then specializes the system for the deployment task without disrupting the carefully learned visual representations.

The decision of whether to freeze the vision encoder during VLM instruction tuning reflects a basic stability-versus-adaptability tradeoff that every VLM practitioner must assess.

Frozen encoders offer significant practical advantages. The encoder's weights occupy several gigabytes of memory, and freezing them allows the optimizer to allocate its memory budget entirely to the language model and projection layer. Training with a frozen encoder is faster per step because no gradients need to be backpropagated through the encoder's dozens of transformer layers. The encoder's visual representations remain stable throughout training, preventing the gradual drift that can occur when the encoder is updated to serve a particular task at the expense of others.

The risk of a frozen encoder emerges when there is a distributional mismatch between the encoder's pretraining data and the deployment domain. A SigLIP encoder trained on web images may produce suboptimal representations for medical scans, satellite imagery, or highly specialized scientific visualizations, because these image types appear rarely or never in web-crawled datasets. In these cases, freezing the encoder locks in poor representations that no amount of projection layer training can fully compensate for.

Fine-tuning the encoder, whether fully or using parameter-efficient methods such as LoRA, allows the representations to adapt to the deployment distribution. Fine-tuning the final few layers of the encoder is a common compromise: the early layers, which learn low-level features like edges and textures that transfer broadly, remain frozen, while the later layers that produce higher-level semantic representations are updated. This selective updating reduces the risk of forgetting low-level visual features while letting adaptation at the semantic level where domain shift matters most.

As we discussed in Catastrophic Forgetting in the context of language models, the same mechanisms of forgetting apply to vision encoders. Aggressive fine-tuning on a narrow dataset can degrade performance on general visual tasks even when in-domain performance improves, because the encoder overwrites broadly useful features with task-specific ones. Regularization techniques such as L2-SP (L2 to Starting Point), which penalizes deviations from the pretrained weights, help mitigate this forgetting while still letting beneficial adaptation.

The projection layer connecting the vision encoder to the language model deserves more attention than it typically receives. While often described as a simple adapter, the projection layer performs several necessary functions beyond dimensional mapping.

The projection layer transforms the statistical distribution of vision embeddings, which are trained to be unit-normalized for cosine similarity, into a distribution that the language model expects for its input token embeddings. Language model token embeddings are typically not unit-normalized and follow a different distribution shaped by the vocabulary pretraining objective. The projection layer must bridge both this distributional gap and the dimensional gap.

Linear projections (a single matrix multiplication) handle dimensional mapping but perform limited distributional transformation. MLP projections with one or two hidden layers and nonlinear activations can reshape the distribution more flexibly, capturing nonlinear relationships between vision and language feature spaces. More recent architectures like Q-Former (used in InstructBLIP) use a small transformer-based query module that learns to extract the most language-relevant information from the full set of patch tokens, effectively performing a learned compression of the visual representation before passing it to the language model.

The projection layer's training dynamics also matter. When trained from scratch with randomly initialized weights, the projection layer must quickly learn to produce sensible outputs for the language model, or early training steps will be noisy. Initializing the projection layer with values that produce language model inputs near the embedding mean, rather than large outliers, leads to more stable early training and faster convergence.