Vision Encoders for VLMs › Encoder Selection Criteria

Selecting a vision encoder for a VLM involves balancing representation quality, computational constraints, and alignment characteristics. The choice of encoder fundamentally constrains what the VLM can perceive and understand, which makes it as necessary as the selection of the language backbone itself.

The distribution and quality of pretraining data strongly affect downstream capabilities:

• LAION-2B and similar web-scale datasets: These provide broad coverage of natural images and objects across scenes but contain noisy alignments between images and text. Encoders trained here excel at natural images but may struggle with structured domains like charts, documents, or specialized scientific imagery.

• Synthetic data, including text rendering and diagrams: Training on rendered text, geometric diagrams, and synthetic documents improves OCR and document understanding capabilities but may reduce performance on natural images by shifting the feature distribution.

• Curated academic datasets such as ImageNet, COCO, Visual Genome offer higher quality per-image annotations and cleaner labels, but their smaller scale potentially limits generalization compared to billion-scale web datasets.

The domain gap between pretraining data and deployment scenarios often determines whether a frozen encoder will suffice or whether domain-specific fine-tuning is necessary. A VLM deployed as a general-purpose assistant answering questions about everyday photographs will work well with any encoder pretrained on web images. A VLM deployed to assist radiologists by interpreting chest X-rays needs either an encoder pretrained on medical images or significant fine-tuning data to bridge the domain gap.

Beyond the distribution of images, the quality of the text paired with images matters enormously. Web alt-text is notoriously inconsistent: some images have descriptive captions that accurately describe the scene, others have keyword-stuffed SEO text, and others have empty or boilerplate captions. Encoders trained on higher-quality captions, such as those generated by a larger vision-language model, often produce better alignment between image and text representations, as the text supervision more accurately reflects the visual content.

Consider your target use cases when selecting resolution:

• 224x224: Sufficient for general scene understanding, object recognition, and coarse-grained tasks. This resolution offers the fastest inference and lowest memory footprint, which makes it suitable for real-time applications or resource-constrained environments.

• 336x336 or 384x384: Better for fine-grained classification, small text recognition, and detailed attribute extraction. These resolutions capture most everyday visual details without excessive computational overhead.

• 448x448 and above: Necessary for document analysis, high-resolution charts, scientific figures, and tasks requiring fine spatial reasoning. The significant computational cost of these resolutions must be justified by the specific requirements of the application domain.

The resolution decision also interacts with the downstream serving infrastructure. Models deployed on consumer hardware, such as a laptop or mobile device, face strict memory constraints that limit the usable resolution. Models served via cloud APIs can afford higher resolutions when latency requirements allow, but per-request costs scale with the number of patches, making high-resolution processing economically significant at scale.

Vision encoders for VLMs adopt different alignment approaches depending on the training pipeline and resource constraints:

• Frozen encoders: The most common approach keeps the vision encoder fixed during VLM training, with alignment learned through projection layers or adapter modules that map vision features into the language model's input space. This approach requires an encoder already well-aligned with language concepts but offers stability and prevents catastrophic forgetting of visual features.

• Partial fine-tuning: Updates only the final layers of the vision encoder during VLM training, letting adaptation to the language backbone while preserving low-level visual features learned during pretraining. This strikes a balance between flexibility and stability.

• Full fine-tuning: Trains the entire encoder alongside the language model. It offers maximum alignment flexibility at the risk of catastrophic forgetting of visual features. This approach often uses parameter-efficient methods such as LoRA to limit the number of updated parameters while still letting adaptation.

The alignment strategy also affects evaluation and benchmarking. Frozen encoders make it straightforward to measure the contribution of the projection layer and language model in isolation, because the vision features are fixed. Fine-tuned encoders entangle the encoder's adaptation with the language model's learning, which makes it harder to attribute performance gains to specific components. For research purposes, starting with a frozen encoder provides cleaner ablations; for production systems, the marginal performance gain from fine-tuning sometimes justifies the additional complexity.

Beyond simple parameter count, consider these operational metrics:

• Patches per second: Throughput at the target resolution, determined by patch embedding efficiency, transformer block optimization, and hardware utilization. This metric directly impacts latency in production systems.

• KV-cache size: For autoregressive VLMs such as GPT-style models, the vision encoder's output is the prefix to text generation. Longer patch sequences increase memory pressure during decoding, as the key and value tensors for all vision tokens must be stored and attended to during every generation step.

• Multi-frame support: Video understanding requires processing multiple frames, sometimes dozens or hundreds per clip. Encoder efficiency compounds linearly with frame count, making efficiency necessary for temporal understanding tasks.

For multi-frame video inputs, the per-frame cost of encoding can dominate total latency and memory usage. A video with 30 frames at 224x224 produces 30 x 196 = 5,880 patch tokens for a 16x16 patch encoder. At 384x384 with 14x14 patches, the same 30 frames produce 30 x 729 = 21,870 tokens. At that scale, the language model's context window becomes the binding constraint, and video understanding systems must either select keyframes or apply temporal compression to reduce the number of frames passed to the language model.