Vision Encoders for VLMs › Limitations and Impact

Vision encoders for VLMs face basic constraints that limit current multimodal capabilities and shape the design space of practical systems. The quadratic scaling of attention with image resolution remains the primary bottleneck, forcing a difficult tradeoff between fine-grained visual understanding and computational feasibility. While techniques like FlashAttention reduce memory overhead for a given resolution through optimized memory access patterns, they do not change the underlying $O(N^2)$ complexity with respect to sequence length. This limitation particularly impacts document understanding and scientific figure analysis, where necessary information may exist at sub-patch scales, such as small text annotations, fine textures, or subtle visual cues that require 896x896 or higher resolution to resolve adequately. Current research directions, including sparse attention patterns for vision and hybrid CNN-transformer architectures, attempt to circumvent this barrier by reducing the effective sequence length or processing only relevant regions, but no solution yet matches the uniform scalability of text-only models where sequence length grows linearly with content.

The domain specificity of vision encoders presents another significant challenge that complicates the development of general-purpose multimodal systems. Encoders pretrained on natural images from sources like LAION or COYO transfer poorly to specialized domains such as medical imaging, satellite photography, industrial inspection, or scientific visualizations without extensive fine-tuning. Unlike language models that benefit from broad pretraining across diverse text domains and often generalize remarkably well to new domains, vision encoders seem more sensitive to the distribution gap between pretraining and deployment data. This sensitivity has led to a fragmentation in the ecosystem where different VLM applications require different vision backbones trained on domain-specific data, complicating the development of unified multimodal systems.

A subtler but important limitation concerns the granularity of visual reasoning that current encoders support. Modern vision encoders, including SigLIP, excel at recognizing objects and scenes as well as visual styles. They struggle with spatial reasoning tasks that require understanding precise geometric relationships, such as counting objects in dense crowds, reasoning about overlapping shapes, or interpreting complex multi-step diagrams. These tasks require the model to maintain and manipulate spatial representations in ways that patch-level attention does not naturally support. The language model can learn to compensate for some of these limitations through instruction tuning, but the basic constraint comes from how the vision encoder discretizes the image into patches that may not align with the semantically meaningful boundaries in the scene.

The choice between frozen and fine-tuned encoders further complicates deployment decisions. Frozen encoders ensure stability, prevent catastrophic forgetting of visual features, and allow efficient training of the language components, but they limit the VLM's ability to adapt to new visual domains or align perfectly with the language backbone's evolving representations during instruction tuning. The tension between stability and adaptability has no universally correct resolution: the best choice depends on the available training data, the deployment domain, and the computational budget available for fine-tuning. Practitioners must evaluate this tradeoff empirically, often discovering that a frozen encoder works sufficiently well for broad natural image tasks but requires at least partial fine-tuning for specialized applications with clear distributional gaps from the pretraining corpus.