Vision Encoders for VLMs: SigLIP, Resolution, Architecture

Vision encoders serve as the necessary bridge between the continuous world of pixels and discrete language tokens, transforming raw image data into dense vector representations that large language models can process. While standalone vision models optimize for specific tasks such as classification, object detection, or segmentation, encoders in Vision-Language Models (VLMs) face a distinct and more complex challenge. They must produce embeddings that capture visual semantics and align precisely with textual representations in a shared embedding space, all while handling the high dimensionality of visual data with computational efficiency.

As we discussed in Vision Transformer and CLIP, the field has largely converged on transformer-based architectures for visual encoding, moving away from convolutional backbones toward patch-based self-attention mechanisms. However, simply repurposing a CLIP vision encoder or standard ViT for a VLM often yields suboptimal results. The specific demands of multimodal systems create unique constraints that differ from standalone vision tasks. These include the need for high-resolution image understanding to capture fine details like small text or textures, efficient cross-modal alignment that remains stable during training dynamics, and inference-time efficiency that does not bottleneck the language generation process. These requirements have driven the development of specialized architectural variants and training objectives specifically engineered for the multimodal context.

This chapter examines the architectural choices that distinguish vision encoders built for VLMs from their standalone counterparts. We will explore SigLIP's sigmoid-based training objective that improves alignment stability and scalability, strategies for handling variable image resolutions without incurring prohibitive quadratic memory costs, and the practical tradeoffs that guide encoder selection for production multimodal systems. By understanding these nuances, you will gain insight into why certain encoders work better for specific VLM applications and how to assess the design space of vision-language integration.

When you integrate vision capabilities into a language model, the encoder must solve three distinct and interconnected problems simultaneously. Each of these challenges arises from the basic differences between how images and text represent information, and solving one often complicates the others.

Dimensional compression. A single 224x224 RGB image contains 150,528 individual pixel values. A VLM encoder must compress this massive, high-dimensional input into a sequence of hundreds of tokens, typically ranging from 256 to 1024 tokens, each with embedding dimensions matching the language model, often between 2048 and 8192 dimensions. This compression must preserve fine-grained spatial details necessary for tasks like optical character recognition or small object detection, while simultaneously capturing high-level semantic concepts required for image captioning or visual question answering. The challenge lies in maintaining both local precision and global coherence within a severely constrained representation budget.

Cross-modal alignment. The output embeddings from the vision encoder must live in the same representational space as the text embeddings produced by the language model's input layers. As we explored in Contrastive Learning for Retrieval, contrastive objectives can align modalities during pretraining, but VLM encoders face the additional challenge of maintaining this alignment when frozen or fine-tuned alongside rapidly evolving language backbones. The vision encoder must produce features that are visually meaningful and semantically compatible with linguistic concepts, letting the language model to reason about visual content using its existing knowledge and reasoning capabilities.

Resolution flexibility. Real-world images vary dramatically in size, aspect ratio, and content density. Unlike text, where tokenization provides natural variable-length sequences and subword units handle different vocabularies gracefully, vision encoders traditionally expect fixed-size inputs. Handling high-resolution inputs of 1024x1024 pixels or higher without exploding memory usage requires architectural innovations. High resolution is often necessary for document understanding, where small text must be legible, or for fine-grained recognition tasks where subtle details determine the correct classification. Yet the quadratic scaling of attention complexity makes naive high-resolution processing computationally prohibitive.

These three problems are deeply intertwined. Improving resolution increases sequence length, which increases the computational cost of attention and limits the batch size available for contrastive alignment. Forcing better compression degrades spatial fidelity. Keeping the encoder frozen maintains alignment stability but prevents adaptation to the language backbone's evolving feature requirements. No current architecture resolves all three problems perfectly, and understanding the tradeoffs is needed for building effective multimodal systems.

The standard approach to vision-language pretraining, exemplified by the original CLIP model, uses a contrastive InfoNCE loss that normalizes similarity scores across the entire batch. While this approach has proven effective for learning joint embeddings, the softmax-based normalization introduces training instabilities and limits scalability as batch sizes grow. SigLIP, which stands for Sigmoid Loss for Language Image Pre-training, offers an alternative that has rapidly become the de facto standard for modern VLM vision encoders, addressing basic limitations in the contrastive learning paradigm.

Recall from Contrastive Learning for Retrieval that InfoNCE computes the probability of matching image-text pairs using a softmax normalization over all possible pairs within the batch:

where:

• $s_{i,j}$: the similarity score between image $i$ and text $j$ (higher values indicate stronger match)
• $s_{i,i}$: the similarity between matching pair $i$ (the positive example)
• $\tau$: temperature parameter controlling the sharpness of the distribution (lower values create sharper distinctions)
• $\sum_{j} \exp(s_{i,j}/\tau)$: normalizing constant summing over all text candidates $j$ in the batch

This formulation computes the log-probability of selecting the correct text match from among all candidates in the batch. The temperature $\tau$ scales the similarities: when $\tau$ approaches 0, the distribution becomes sharp, approaching a one-hot selection of the maximum similarity; when $\tau$ approaches infinity, all probabilities become uniform.

However, the denominator requires computing similarities across the entire batch, creating a synchronization point in distributed training and potential instability when batch sizes grow large. The softmax normalization forces a competitive dynamic where improving the score of one positive pair necessarily affects the gradients of all other pairs in the batch. This global competition means that the gradient signal for any single pair depends on all other pairs in the batch, creating complex training dynamics that are difficult to control at scale.

SigLIP replaces this competitive normalization with a simple binary classification objective. Treating each image-text pair as an independent binary classification problem (match versus no-match), the loss becomes:

where:

• $\mathbb{1}_{i=j}$: indicator function equal to 1 when image $i$ matches text $j$, 0 otherwise (identifies positive pairs)
• $\mathbb{1}_{i \neq j}$: indicator function equal to 1 when image $i$ does not match text $j$ (identifies negative pairs)
• $\sigma(\cdot)$: the sigmoid function $\frac{1}{1 + e^{-s}}$, mapping similarity scores to $(0, 1)$ probability range
• $s_{i,j}$: similarity score between image $i$ and text $j$
• $\sum_{i,j}$: sum over all image-text pairs in the batch

Unlike InfoNCE, this formulation treats each pair as an independent binary classification problem. The sigmoid outputs the probability that a given pair matches, with no normalization across other pairs required. This independence offers three key advantages for VLM encoders:

1. No batch normalization: Each pair's loss is independent, eliminating the costly softmax normalization across devices and letting larger effective batch sizes through gradient accumulation without synchronization barriers. In distributed training, this means each GPU can process its local batch without waiting for global statistics.

2. Stable gradients: The sigmoid's bounded output prevents the extreme gradient magnitudes that can occur with exponentials in large batches. When similarity scores become large in magnitude, the sigmoid saturates gracefully, whereas softmax can produce numerically unstable gradients.

3. Tunable bias: SigLIP introduces a learnable bias term $b$ in the scoring function $s_{i,j} = \langle v_i, t_j \rangle + b$, letting the model to learn an optimal decision threshold rather than assuming zero-centered similarities. This proves particularly valuable when the distribution of positive and negative pairs is imbalanced or when the model needs to adjust its confidence calibration.

One of the more subtle but important details in SigLIP's design is the treatment of its scaling parameters. In the CLIP formulation, the temperature $\tau$ is a learned scalar that controls the sharpness of the probability distribution. In SigLIP, two learnable scalars take its place: a log-temperature $\log t$ and a bias $b$. The full scoring formula becomes:

where:

• $t$: learned temperature, initialized to a value that produces reasonable initial logit magnitudes (often $\exp(10)$ for image-text settings)
• $\langle v_i, t_j \rangle$: cosine similarity between normalized image embedding $v_i$ and text embedding $t_j$
• $b$: learned bias, initialized to a small negative value such as $-10$ to prevent the model from predicting all pairs as matches at the start of training

The negative bias initialization is conceptually important. At the beginning of training, random embeddings produce near-zero cosine similarities. Without a bias, $\sigma(0) = 0.5$, meaning every pair starts with a 50% match probability. For a batch of 1024 images paired with 1024 texts, more than one million pairs exist, and only 1024 of them are true matches. Initializing the bias to $-10$ shifts the sigmoid output toward near-zero for all pairs. This provides a sensible prior that most pairs do not match. The model then learns to push true matches above the threshold while keeping false pairs below it, rather than starting from a confused state where everything looks like a match.

The temperature $t$ serves a similar calibration role. A high initial temperature means the model applies little scaling to similarities, creating soft predictions that allow gradient flow even for near-zero similarities early in training. As training progresses and the encoder produces more discriminative embeddings, the effective temperature can be tuned to sharpen predictions where the model is confident and maintain softness where ambiguity exists.

Despite the different training objective, SigLIP uses the same Vision Transformer architecture as CLIP for the vision encoder. The improvements come primarily from the training dynamics and the shape of the loss surface rather than structural changes. However, SigLIP-trained encoders tend to produce embeddings with better calibration for downstream VLM use, where the vision encoder often remains frozen while the language model adapts to process visual inputs.

The sigmoid loss particularly benefits scenarios with noisy training data, which is common in large-scale web datasets like LAION or COYO-700M. Because SigLIP does not force hard contrasts between all pairs in a batch, it can handle ambiguous pairs that might be weak matches, such as an image that contains multiple concepts or text descriptions that partially match the visual content. This flexibility allows the encoder to preserve fine-grained semantic relationships that competitive normalization might collapse. When an image depicts both a dog and a park bench, an InfoNCE model must decide which of these descriptions is the correct match and push all others toward zero probability. A SigLIP model can assign moderate probability to both the dog description and the park bench description without loss divergence, capturing the ambiguity in the scene.

In practice, the most widely deployed SigLIP configuration is the SigLIP-SO400M family, where SO stands for "Scaled-up by Oversized" and 400M refers to the approximately 400 million parameter vision encoder. This family has become the default vision backbone in several prominent VLMs including PaLI-Gemma, LLaVA successors, and various open-source multimodal systems.

The SigLIP-SO400M-14/384 configuration uses 14x14 patches at 384x384 resolution, creating 729 patch tokens per image. This configuration represents a deliberate compromise: patch size 14 offers finer granularity than the original ViT-B/16 configuration, and the 384x384 resolution captures enough detail for most practical tasks without the memory costs of 512x512 or higher. The "SO" scaling procedure increases the model's depth and width beyond the standard ViT-L specification by applying compound scaling, distributing additional capacity across attention heads, hidden dimensions, and the number of transformer layers simultaneously.

The embeddings produced by SigLIP-SO400M at 384x384 resolution support a remarkably broad range of visual tasks out of the box. Natural image understanding, chart and diagram interpretation, and basic document reading all work well with this configuration as a frozen encoder. This breadth reflects the advantages of training on diverse, web-scale image-text pairs with a stable loss function that preserves multi-concept relationships rather than forcing each image into a single semantic category.

Standard Vision Transformers require careful adaptation for effective VLM integration. The following architectural variants address specific multimodal requirements that arise when vision and language must interact smoothly.

The choice of patch size fundamentally trades off between sequence length, which directly impacts computational cost, and spatial granularity, which determines the level of detail the model can perceive:

• 14x14 patches: Used in high-resolution encoders such as SigLIP-SO400M at 384x384 resolution. This configuration yields finer spatial detail but produces longer sequences. At 384x384 resolution, this yields approximately 27 patches per dimension, or 729 total patches. This granularity supports tasks requiring fine detail, such as reading small text or recognizing subtle textures.

• 16x16 patches: The original ViT standard and still widely used. At 224x224 resolution, this produces 196 patches, a manageable sequence length for most LLM attention mechanisms. This represents a balanced compromise between detail and efficiency.

• 32x32 patches: Aggressive compression for efficiency-focused applications. This produces only 49 patches at 224x224 resolution, letting faster inference and lower memory usage but potentially losing fine details necessary for text recognition, small object detection, or precise spatial localization.

Patch size affects more than sequence length management. Smaller patches preserve more spatial information per token, which helps when the model needs to reason about the relative positions and spatial relationships of objects. Consider a task like reading a table in a document: each cell must be individually recognizable and its position within the table must be understood. With 32x32 patches, an entire table row might collapse into a single patch, destroying the cell boundaries that give the table structure. With 14x14 patches, individual characters within cells may be readable, depending on font size and resolution.

While we explored various position encoding schemes in Part XIV: Positional Encoding, images present unique requirements that differ fundamentally from sequential text. Unlike text, which has a one-dimensional linear structure, images possess two-dimensional spatial structure that must be preserved for the model to understand spatial relationships, object locations, and visual layout.

2D Sinusoidal encodings extend the 1D sinusoidal approach to two dimensions, creating position representations that capture both horizontal and vertical coordinates:

where:

• $(x, y)$: spatial coordinates of the patch in the 2D image grid (integers representing row and column indices)
• $d$: total embedding dimension of the model
• $k$: frequency index ranging from $0$ to $d/4 - 1$, controlling the wavelength of the encoding
• $PE(x, y, 2k)$: even-indexed dimensions of the position encoding vector, using sine functions
• $PE(x, y, 2k+1)$: odd-indexed dimensions, using cosine functions
• $10000^{4k/d}$: wavelength scaling factor that creates different frequencies for different dimensions

The product of sine and cosine terms encodes both horizontal and vertical position simultaneously, letting the model to attend to specific spatial locations. Lower values of $k$ produce slowly-varying encodings capturing coarse position and general layout, while higher $k$ produce rapid oscillations for fine-grained localization and precise spatial reasoning.

Learned 2D embeddings treat row and column positions separately, adding two distinct embedding vectors per patch. This approach offers more flexibility than fixed sinusoidal patterns, as the model can learn position representations optimized for its specific task. However, learned embeddings require fixed grid sizes unless using interpolation techniques, and they may not generalize as well to resolutions unseen during training.

Absolute position bias approaches, similar to ALiBi discussed in ALiBi, apply position-based attention biases rather than adding to embeddings. This method adds a bias term to attention scores based on the distance between patches, potentially improving generalization to resolutions unseen during training and reducing the need for explicit position embeddings.

The relative merits of these approaches depend heavily on the downstream VLM application. For tasks where exact spatial coordinates matter, such as object grounding where you need to predict bounding box coordinates, sinusoidal encodings provide consistent spatial reference frames that transfer more naturally to coordinate prediction. For tasks where broad semantic understanding matters more than precise localization, learned embeddings often yield better representations because they can capture task-relevant spatial relationships that fixed encodings may miss. Many modern VLM encoders use learned 2D embeddings with bicubic interpolation for resolution adaptation, accepting the slight cost of interpolation artifacts in exchange for the flexibility of learned representations.

Standard ViT architectures use a CLS token for global aggregation, prepending a special token to the sequence and using its final representation as the global image embedding. However, VLMs often benefit from attention pooling, where the final representation is computed as a weighted sum of patch tokens:

where:

• $\text{output}$: final aggregated vision representation fed to the language model
• $\alpha_i$: learned attention weight for patch $i$, computed via multi-head attention with a learnable query vector (indicates importance of each patch)
• $v_i$: embedding vector for the $i$-th image patch
• $\sum_{i}$: weighted sum over all patches in the image

This formulation allows dynamic, content-dependent aggregation where the model learns which image regions to emphasize based on their relevance. Unlike a fixed CLS token aggregation, which must compress all information into a single vector regardless of content, attention pooling can focus on task-relevant regions. For instance, the model might emphasize text regions when performing OCR, facial regions when identifying people, or object boundaries during fine-grained recognition. This dynamic selection allows the encoder to adapt its output based on the visual content. This provides more informative representations to the language model than a fixed aggregation strategy could achieve.

Beyond single-vector pooling, many modern VLMs pass the full sequence of patch tokens to the language model rather than pooling at all. This approach, used in architectures like LLaVA and PaLI-Gemma, allows the language model to perform its own implicit spatial reasoning by attending to different patches depending on the question asked. The language model can attend to the patch tokens where the relevant answer appears, rather than relying on the vision encoder to anticipate which regions matter. This design trades computational efficiency for flexibility: passing all 729 patches from a SigLIP-SO400M-384 encoder to a 7B language model creates a prefix of 729 tokens that the language model must attend to on every generation step.

Resolution handling represents one of the most active and challenging areas of VLM encoder research. The quadratic memory complexity of self-attention with respect to sequence length makes high-resolution processing computationally expensive, yet many practical tasks, including document understanding, scientific figure analysis, and fine-grained recognition, require pixel-level detail that only high resolution can provide.

For a ViT with patch size $p$ processing an image of height $h$ and width $w$, the sequence length is determined by:

where:

• $N$: total number of patches (sequence length) fed to the transformer
• $h$: image height in pixels
• $w$: image width in pixels
• $p$: patch size (square patches assumed, so $p \times p$ pixels per patch)
• $\left\lceil \cdot \right\rceil$: ceiling function so partial patches are included

This formula demonstrates how image resolution translates directly to sequence length. Doubling both height and width quadruples $N$, which quadruples the attention memory since self-attention complexity scales with $O(N^2 \cdot d)$, where $d$ is the embedding dimension. Doubling image resolution while keeping patch size constant increases the number of patches by a factor of four, resulting in a sixteen-fold increase in attention computation.

Consider practical examples:

• 224x224 with 14x14 patches: 256 patches, manageable for most hardware
• 448x448 with 14x14 patches: 1,024 patches, requiring 4x the memory of 224px
• 896x896 with 14x14 patches: 4,096 patches, requiring 16x the memory of 224px

This quadratic scaling means that moving to higher resolutions quickly exhausts GPU memory, even with modern hardware, creating a basic tension between the need for fine detail and computational feasibility.

The memory cost compounds further in the context of a full VLM. The vision encoder processes the image once to produce patch tokens. These tokens then become part of the language model's context, and the language model must attend to all of them on every generation step during inference. A 7B parameter language model generating a response to a question about a high-resolution document image at 896x896 with 14x14 patches must attend to 4,096 vision tokens on top of the text tokens for every step of generation. On typical hardware, this quickly saturates the available memory budget for KV cache entries, severely limiting batch sizes or maximum output lengths.

Rather than resizing all images to a fixed square and distorting aspect ratios, modern VLM encoders often support native resolution processing, which maintains the original image proportions:

1. Dynamic padding: Images maintain their aspect ratio, with shorter sides padded with special padding tokens or zeros to match the required patch grid alignment. This prevents distortion but requires handling variable sequence lengths.

2. Variable sequence lengths: The encoder processes different image sizes with different sequence lengths, requiring the VLM to handle variable-length visual inputs. This is often managed through position-aware attention masking that indicates which tokens are real content versus padding.

3. Aspect ratio bucketing: During training, images are grouped into buckets with similar aspect ratios, such as 1:1 for portraits, 4:3 for standard photography, or 16:9 for widescreen images. This minimizes padding waste by so images in the same batch have consistent dimensions, then processes each bucket separately.

Maintaining native resolution and aspect ratio matters more for some content types than others. A photograph of a person's face looks natural when resized to a square because the face is roughly symmetric. A panoramic photograph distorted to a square might still be recognizable for scene understanding. But a document or spreadsheet distorted to a square becomes much harder to read: column alignments shift, text aspect ratios change, and tabular structure becomes ambiguous. For VLMs intended to handle business documents, invoices, or structured reports, native resolution processing is a correctness requirement rather than a quality enhancement.

When single-image processing hits memory limits, tiling strategies, also known as patch merging or sliding window approaches, divide high-resolution images into smaller, manageable crops:

• Non-overlapping tiles: Split the image into independent regions that are processed separately. While this approach is memory-efficient, it loses cross-tile context and may cut objects at tile boundaries, potentially missing relationships between objects in different tiles or fragmenting large objects.

• Overlapping tiles with global context: Process the full image at reduced resolution to capture global context, alongside high-resolution crops for local detail. The VLM learns to fuse these representations, combining the spatial precision of high-resolution crops with the semantic coherence of the global view. However, this approach doubles encoding cost since each image must be processed twice.

• Multi-scale feature pyramids: Process the image at multiple resolutions simultaneously, combining features from different scales before feeding to the language model. This resembles the hierarchical approaches in Sliding Window Attention but applies them to vision resolution, letting the model to attend to both fine details and global structure without prohibitive memory costs.

The overlapping tiles with global context strategy, used in models like LLaVA-HR and InternVL, has become particularly popular. It works by first processing the image at a standard low resolution, such as 224x224 or 336x336, to obtain a global understanding of the scene. Then the image is divided into a 2x2 or 3x3 grid of high-resolution tiles, each processed independently. The global representation provides context that prevents the model from misinterpreting tile-level details, while the tile representations provide the fine-grained spatial information needed for tasks like reading small text. A model asked whether a chart shows an upward or downward trend uses the global view to locate the chart axes and overall shape, then the high-resolution tiles to read the precise numerical labels on those axes.

Just as Position Interpolation extends language models to longer contexts beyond their training length, vision encoders can extend to higher resolutions through position encoding interpolation.

If a model was trained on 224x224 images, creating a 14x14 grid of 16x16 patches, extending to 448x448 resolution produces a 28x28 grid. This requires interpolating the learned position embeddings from the original 14x14 grid to the new 28x28 grid. Linear or bicubic interpolation works reasonably well for this expansion, though fine-tuning on higher resolution images significantly improves performance by letting the model to adapt to the new spatial frequencies and details available at higher resolution.

The interpolation quality depends significantly on how the position embeddings were originally learned. Embeddings trained with a smooth supervision signal, such as by minimizing reconstruction loss on position-perturbed images, interpolate more gracefully than embeddings learned purely from downstream task gradients. This is analogous to the observation in language models that position encodings trained with NTK-aware scaling interpolate more smoothly than those trained without attention to extrapolation properties, as we discussed in NTK-Aware Scaling.

Understanding how vision encoders are trained and adapted for VLMs is needed for making good architectural choices. The interplay between the encoder, the projection layer, and the language model during training determines how well the final system generalizes across visual tasks.

Most production VLMs train their vision components in multiple stages, each serving a distinct purpose:

Stage 1: Vision encoder pretraining. The vision encoder is trained independently on a large image-text dataset using a contrastive objective such as SigLIP or InfoNCE. At this stage, the language backbone is not involved. The encoder learns to produce image embeddings that align with text embeddings from a separate text encoder. The scale of this pretraining is enormous: SigLIP-SO400M was trained on roughly 4 billion image-text pairs, giving it broad visual coverage before any multimodal instruction tuning occurs.

Stage 2: Cross-modal projection alignment. A lightweight projection layer, typically a two-layer MLP or a linear projection, is trained to map vision encoder outputs into the language model's token embedding space. During this stage, both the vision encoder and language backbone are usually frozen, and only the projection layer is updated. The projection layer learns to translate the statistical properties of vision embeddings into a distribution that the language model can interpret as meaningful token-like inputs. This stage is computationally cheap relative to the full training, requiring only a few hundred million to a few billion image-text pairs.

Stage 3: Instruction tuning. The full system is fine-tuned on instruction-following data that pairs images with questions and answers. At this stage, the language model is updated to learn how to use visual inputs to generate helpful responses. The vision encoder may remain frozen, or a small subset of its final layers may be unfrozen to allow limited adaptation. This stage uses far less data than pretraining, typically tens of millions of examples, but requires high-quality annotations.

This three-stage approach reflects a principled decomposition of the learning problem. Pretraining the encoder separately ensures it develops reliable visual representations before the noise introduced by language model gradients can interfere. Projection alignment ensures the modalities can communicate before the full model is trained to follow instructions. Instruction tuning then specializes the system for the deployment task without disrupting the carefully learned visual representations.

The decision of whether to freeze the vision encoder during VLM instruction tuning reflects a basic stability-versus-adaptability tradeoff that every VLM practitioner must assess.

Frozen encoders offer significant practical advantages. The encoder's weights occupy several gigabytes of memory, and freezing them allows the optimizer to allocate its memory budget entirely to the language model and projection layer. Training with a frozen encoder is faster per step because no gradients need to be backpropagated through the encoder's dozens of transformer layers. The encoder's visual representations remain stable throughout training, preventing the gradual drift that can occur when the encoder is updated to serve a particular task at the expense of others.

The risk of a frozen encoder emerges when there is a distributional mismatch between the encoder's pretraining data and the deployment domain. A SigLIP encoder trained on web images may produce suboptimal representations for medical scans, satellite imagery, or highly specialized scientific visualizations, because these image types appear rarely or never in web-crawled datasets. In these cases, freezing the encoder locks in poor representations that no amount of projection layer training can fully compensate for.

Fine-tuning the encoder, whether fully or using parameter-efficient methods such as LoRA, allows the representations to adapt to the deployment distribution. Fine-tuning the final few layers of the encoder is a common compromise: the early layers, which learn low-level features like edges and textures that transfer broadly, remain frozen, while the later layers that produce higher-level semantic representations are updated. This selective updating reduces the risk of forgetting low-level visual features while letting adaptation at the semantic level where domain shift matters most.

As we discussed in Catastrophic Forgetting in the context of language models, the same mechanisms of forgetting apply to vision encoders. Aggressive fine-tuning on a narrow dataset can degrade performance on general visual tasks even when in-domain performance improves, because the encoder overwrites broadly useful features with task-specific ones. Regularization techniques such as L2-SP (L2 to Starting Point), which penalizes deviations from the pretrained weights, help mitigate this forgetting while still letting beneficial adaptation.

The projection layer connecting the vision encoder to the language model deserves more attention than it typically receives. While often described as a simple adapter, the projection layer performs several necessary functions beyond dimensional mapping.

The projection layer transforms the statistical distribution of vision embeddings, which are trained to be unit-normalized for cosine similarity, into a distribution that the language model expects for its input token embeddings. Language model token embeddings are typically not unit-normalized and follow a different distribution shaped by the vocabulary pretraining objective. The projection layer must bridge both this distributional gap and the dimensional gap.

Linear projections (a single matrix multiplication) handle dimensional mapping but perform limited distributional transformation. MLP projections with one or two hidden layers and nonlinear activations can reshape the distribution more flexibly, capturing nonlinear relationships between vision and language feature spaces. More recent architectures like Q-Former (used in InstructBLIP) use a small transformer-based query module that learns to extract the most language-relevant information from the full set of patch tokens, effectively performing a learned compression of the visual representation before passing it to the language model.

The projection layer's training dynamics also matter. When trained from scratch with randomly initialized weights, the projection layer must quickly learn to produce sensible outputs for the language model, or early training steps will be noisy. Initializing the projection layer with values that produce language model inputs near the embedding mean, rather than large outliers, leads to more stable early training and faster convergence.

Selecting a vision encoder for a VLM involves balancing representation quality, computational constraints, and alignment characteristics. The choice of encoder fundamentally constrains what the VLM can perceive and understand, which makes it as necessary as the selection of the language backbone itself.

The distribution and quality of pretraining data strongly affect downstream capabilities:

• LAION-2B and similar web-scale datasets: These provide broad coverage of natural images and objects across scenes but contain noisy alignments between images and text. Encoders trained here excel at natural images but may struggle with structured domains like charts, documents, or specialized scientific imagery.

• Synthetic data, including text rendering and diagrams: Training on rendered text, geometric diagrams, and synthetic documents improves OCR and document understanding capabilities but may reduce performance on natural images by shifting the feature distribution.

• Curated academic datasets such as ImageNet, COCO, Visual Genome offer higher quality per-image annotations and cleaner labels, but their smaller scale potentially limits generalization compared to billion-scale web datasets.

The domain gap between pretraining data and deployment scenarios often determines whether a frozen encoder will suffice or whether domain-specific fine-tuning is necessary. A VLM deployed as a general-purpose assistant answering questions about everyday photographs will work well with any encoder pretrained on web images. A VLM deployed to assist radiologists by interpreting chest X-rays needs either an encoder pretrained on medical images or significant fine-tuning data to bridge the domain gap.

Beyond the distribution of images, the quality of the text paired with images matters enormously. Web alt-text is notoriously inconsistent: some images have descriptive captions that accurately describe the scene, others have keyword-stuffed SEO text, and others have empty or boilerplate captions. Encoders trained on higher-quality captions, such as those generated by a larger vision-language model, often produce better alignment between image and text representations, as the text supervision more accurately reflects the visual content.

Consider your target use cases when selecting resolution:

• 224x224: Sufficient for general scene understanding, object recognition, and coarse-grained tasks. This resolution offers the fastest inference and lowest memory footprint, which makes it suitable for real-time applications or resource-constrained environments.

• 336x336 or 384x384: Better for fine-grained classification, small text recognition, and detailed attribute extraction. These resolutions capture most everyday visual details without excessive computational overhead.

• 448x448 and above: Necessary for document analysis, high-resolution charts, scientific figures, and tasks requiring fine spatial reasoning. The significant computational cost of these resolutions must be justified by the specific requirements of the application domain.

The resolution decision also interacts with the downstream serving infrastructure. Models deployed on consumer hardware, such as a laptop or mobile device, face strict memory constraints that limit the usable resolution. Models served via cloud APIs can afford higher resolutions when latency requirements allow, but per-request costs scale with the number of patches, making high-resolution processing economically significant at scale.

Vision encoders for VLMs adopt different alignment approaches depending on the training pipeline and resource constraints:

• Frozen encoders: The most common approach keeps the vision encoder fixed during VLM training, with alignment learned through projection layers or adapter modules that map vision features into the language model's input space. This approach requires an encoder already well-aligned with language concepts but offers stability and prevents catastrophic forgetting of visual features.

• Partial fine-tuning: Updates only the final layers of the vision encoder during VLM training, letting adaptation to the language backbone while preserving low-level visual features learned during pretraining. This strikes a balance between flexibility and stability.

• Full fine-tuning: Trains the entire encoder alongside the language model. It offers maximum alignment flexibility at the risk of catastrophic forgetting of visual features. This approach often uses parameter-efficient methods such as LoRA to limit the number of updated parameters while still letting adaptation.

The alignment strategy also affects evaluation and benchmarking. Frozen encoders make it straightforward to measure the contribution of the projection layer and language model in isolation, because the vision features are fixed. Fine-tuned encoders entangle the encoder's adaptation with the language model's learning, which makes it harder to attribute performance gains to specific components. For research purposes, starting with a frozen encoder provides cleaner ablations; for production systems, the marginal performance gain from fine-tuning sometimes justifies the additional complexity.

Beyond simple parameter count, consider these operational metrics:

• Patches per second: Throughput at the target resolution, determined by patch embedding efficiency, transformer block optimization, and hardware utilization. This metric directly impacts latency in production systems.

• KV-cache size: For autoregressive VLMs such as GPT-style models, the vision encoder's output is the prefix to text generation. Longer patch sequences increase memory pressure during decoding, as the key and value tensors for all vision tokens must be stored and attended to during every generation step.

• Multi-frame support: Video understanding requires processing multiple frames, sometimes dozens or hundreds per clip. Encoder efficiency compounds linearly with frame count, making efficiency necessary for temporal understanding tasks.

For multi-frame video inputs, the per-frame cost of encoding can dominate total latency and memory usage. A video with 30 frames at 224x224 produces 30 x 196 = 5,880 patch tokens for a 16x16 patch encoder. At 384x384 with 14x14 patches, the same 30 frames produce 30 x 729 = 21,870 tokens. At that scale, the language model's context window becomes the binding constraint, and video understanding systems must either select keyframes or apply temporal compression to reduce the number of frames passed to the language model.

Let us calculate the computational implications of different resolution choices for a VLM vision encoder to understand the practical constraints facing system designers.

Consider a ViT-L/14 encoder, which is the Large variant with 14x14 patches and 1024 hidden dimensions, processing images at various resolutions:

The table reveals why resolution scaling presents such a formidable challenge in VLM design. Moving from 224px to 448px increases attention memory by 16x, from approximately 3 MB to 50 MB per layer, and increases compute by 16x as well. For a 24-layer ViT-L, this represents the difference between fitting comfortably on a consumer GPU and requiring enterprise-grade hardware with substantial memory capacity.

Notice that increasing patch size from 14 to 16 reduces sequence length by 23%, from 256 patches to 196 patches. It offers a simple efficiency lever at the cost of spatial precision. This tradeoff between patch size and resolution represents a basic design decision in vision encoder architecture. The SigLIP Base configuration at 384x384 achieves a sequence length of 729 with a 768-dimensional embedding, compared to 1024 dimensions in the ViT-L configurations. The lower dimensionality helps offset the higher patch count. This shows how careful co-design of patch size and resolution alongside embedding dimension can achieve a better computational footprint than tuning any single parameter in isolation.

Let us implement a complete pipeline for processing images through different vision encoder configurations, comparing CLIP-style and SigLIP-style processing to understand the practical differences in training objectives.

Now let us demonstrate how to adapt a pre-trained encoder for higher resolution using position interpolation, a technique that allows models to process images larger than their training resolution without requiring complete retraining from scratch:

Let us compare the output distributions between standard contrastive training and SigLIP-style normalization to observe how the different loss functions affect the probability distribution:

The demonstration illustrates how SigLIP treats each image-text pair as an independent prediction. This allows multiple pairs to have high probabilities simultaneously if they are all valid matches. In contrast, CLIP forces competition between pairs through softmax normalization, requiring the model to choose a single best match even when multiple valid matches exist. This independence allows SigLIP to express uncertainty about multiple potential matches without the hard constraint that probabilities must sum to 1. This provides more flexible training dynamics especially with noisy or ambiguous data.

The off-diagonal probabilities in the SigLIP output are not a sign of failure: they reflect the overlap between image-text pairs that share similar concepts. An image of a dog playing in a park has partial correspondence with text descriptions about parks and about dogs independently. SigLIP's loss function rewards the model for assigning high probability to the correct pair while still letting moderate probabilities for semantically related pairs, whereas CLIP's softmax forcibly suppresses all off-diagonal values toward zero regardless of their semantic relationship to the diagonal entry.

Vision encoders for VLMs face basic constraints that limit current multimodal capabilities and shape the design space of practical systems. The quadratic scaling of attention with image resolution remains the primary bottleneck, forcing a difficult tradeoff between fine-grained visual understanding and computational feasibility. While techniques like FlashAttention reduce memory overhead for a given resolution through optimized memory access patterns, they do not change the underlying $O(N^2)$ complexity with respect to sequence length. This limitation particularly impacts document understanding and scientific figure analysis, where necessary information may exist at sub-patch scales, such as small text annotations, fine textures, or subtle visual cues that require 896x896 or higher resolution to resolve adequately. Current research directions, including sparse attention patterns for vision and hybrid CNN-transformer architectures, attempt to circumvent this barrier by reducing the effective sequence length or processing only relevant regions, but no solution yet matches the uniform scalability of text-only models where sequence length grows linearly with content.

The domain specificity of vision encoders presents another significant challenge that complicates the development of general-purpose multimodal systems. Encoders pretrained on natural images from sources like LAION or COYO transfer poorly to specialized domains such as medical imaging, satellite photography, industrial inspection, or scientific visualizations without extensive fine-tuning. Unlike language models that benefit from broad pretraining across diverse text domains and often generalize remarkably well to new domains, vision encoders seem more sensitive to the distribution gap between pretraining and deployment data. This sensitivity has led to a fragmentation in the ecosystem where different VLM applications require different vision backbones trained on domain-specific data, complicating the development of unified multimodal systems.

A subtler but important limitation concerns the granularity of visual reasoning that current encoders support. Modern vision encoders, including SigLIP, excel at recognizing objects and scenes as well as visual styles. They struggle with spatial reasoning tasks that require understanding precise geometric relationships, such as counting objects in dense crowds, reasoning about overlapping shapes, or interpreting complex multi-step diagrams. These tasks require the model to maintain and manipulate spatial representations in ways that patch-level attention does not naturally support. The language model can learn to compensate for some of these limitations through instruction tuning, but the basic constraint comes from how the vision encoder discretizes the image into patches that may not align with the semantically meaningful boundaries in the scene.

The choice between frozen and fine-tuned encoders further complicates deployment decisions. Frozen encoders ensure stability, prevent catastrophic forgetting of visual features, and allow efficient training of the language components, but they limit the VLM's ability to adapt to new visual domains or align perfectly with the language backbone's evolving representations during instruction tuning. The tension between stability and adaptability has no universally correct resolution: the best choice depends on the available training data, the deployment domain, and the computational budget available for fine-tuning. Practitioners must evaluate this tradeoff empirically, often discovering that a frozen encoder works sufficiently well for broad natural image tasks but requires at least partial fine-tuning for specialized applications with clear distributional gaps from the pretraining corpus.

Vision encoders for VLMs extend standard Vision Transformers to meet the specific demands of multimodal integration, serving as the necessary interface between visual perception and linguistic reasoning. Key developments include SigLIP's sigmoid-based training objective, which improves alignment stability by replacing batch-normalized contrastive loss with independent binary classification, thereby letting more scalable and stable training on large, noisy datasets. The learnable temperature and bias parameters in SigLIP provide calibration flexibility that helps the model handle the extreme imbalance between positive and negative pairs in large-scale training, a property that becomes increasingly important as dataset scale grows into the billions of examples.

Flexible resolution handling strategies, including native resolution processing, aspect ratio bucketing, and position interpolation, balance computational cost against spatial precision, letting VLMs to process everything from standard photographs to high-resolution documents. The selection of patch size, position encoding scheme, and resolution capabilities fundamentally constrains what a VLM can perceive and understand, making encoder architecture a necessary design decision alongside the choice of language backbone. The three-stage training pipeline separates encoder pretraining, cross-modal alignment, and instruction tuning into well-defined phases that allow each component to learn its role without interference from the others.

As multimodal systems continue to evolve, the boundary between vision and language processing continues to blur, with vision encoders taking on increasingly sophisticated roles in joint reasoning tasks. The techniques explored in this chapter, particularly native resolution processing and efficient attention mechanisms, enable the next generation of VLMs to handle high-resolution inputs such as documents and diagrams alongside detailed photographs with high fidelity. In the next chapter, we will explore how these vision encoder outputs connect to language models through projection layers and cross-modal alignment mechanisms, completing the bridge from pixels to language and supporting integrated multimodal reasoning.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about vision encoders for Vision-Language Models.