Vision Encoders for VLMs › ViT Variants for Multimodal Systems

Standard Vision Transformers require careful adaptation for effective VLM integration. The following architectural variants address specific multimodal requirements that arise when vision and language must interact smoothly.

The choice of patch size fundamentally trades off between sequence length, which directly impacts computational cost, and spatial granularity, which determines the level of detail the model can perceive:

• 14x14 patches: Used in high-resolution encoders such as SigLIP-SO400M at 384x384 resolution. This configuration yields finer spatial detail but produces longer sequences. At 384x384 resolution, this yields approximately 27 patches per dimension, or 729 total patches. This granularity supports tasks requiring fine detail, such as reading small text or recognizing subtle textures.

• 16x16 patches: The original ViT standard and still widely used. At 224x224 resolution, this produces 196 patches, a manageable sequence length for most LLM attention mechanisms. This represents a balanced compromise between detail and efficiency.

• 32x32 patches: Aggressive compression for efficiency-focused applications. This produces only 49 patches at 224x224 resolution, letting faster inference and lower memory usage but potentially losing fine details necessary for text recognition, small object detection, or precise spatial localization.

Patch size affects more than sequence length management. Smaller patches preserve more spatial information per token, which helps when the model needs to reason about the relative positions and spatial relationships of objects. Consider a task like reading a table in a document: each cell must be individually recognizable and its position within the table must be understood. With 32x32 patches, an entire table row might collapse into a single patch, destroying the cell boundaries that give the table structure. With 14x14 patches, individual characters within cells may be readable, depending on font size and resolution.

While we explored various position encoding schemes in Part XIV: Positional Encoding, images present unique requirements that differ fundamentally from sequential text. Unlike text, which has a one-dimensional linear structure, images possess two-dimensional spatial structure that must be preserved for the model to understand spatial relationships, object locations, and visual layout.

2D Sinusoidal encodings extend the 1D sinusoidal approach to two dimensions, creating position representations that capture both horizontal and vertical coordinates:

where:

• $(x, y)$: spatial coordinates of the patch in the 2D image grid (integers representing row and column indices)
• $d$: total embedding dimension of the model
• $k$: frequency index ranging from $0$ to $d/4 - 1$, controlling the wavelength of the encoding
• $PE(x, y, 2k)$: even-indexed dimensions of the position encoding vector, using sine functions
• $PE(x, y, 2k+1)$: odd-indexed dimensions, using cosine functions
• $10000^{4k/d}$: wavelength scaling factor that creates different frequencies for different dimensions

The product of sine and cosine terms encodes both horizontal and vertical position simultaneously, letting the model to attend to specific spatial locations. Lower values of $k$ produce slowly-varying encodings capturing coarse position and general layout, while higher $k$ produce rapid oscillations for fine-grained localization and precise spatial reasoning.

Learned 2D embeddings treat row and column positions separately, adding two distinct embedding vectors per patch. This approach offers more flexibility than fixed sinusoidal patterns, as the model can learn position representations optimized for its specific task. However, learned embeddings require fixed grid sizes unless using interpolation techniques, and they may not generalize as well to resolutions unseen during training.

Absolute position bias approaches, similar to ALiBi discussed in ALiBi, apply position-based attention biases rather than adding to embeddings. This method adds a bias term to attention scores based on the distance between patches, potentially improving generalization to resolutions unseen during training and reducing the need for explicit position embeddings.

The relative merits of these approaches depend heavily on the downstream VLM application. For tasks where exact spatial coordinates matter, such as object grounding where you need to predict bounding box coordinates, sinusoidal encodings provide consistent spatial reference frames that transfer more naturally to coordinate prediction. For tasks where broad semantic understanding matters more than precise localization, learned embeddings often yield better representations because they can capture task-relevant spatial relationships that fixed encodings may miss. Many modern VLM encoders use learned 2D embeddings with bicubic interpolation for resolution adaptation, accepting the slight cost of interpolation artifacts in exchange for the flexibility of learned representations.

Standard ViT architectures use a CLS token for global aggregation, prepending a special token to the sequence and using its final representation as the global image embedding. However, VLMs often benefit from attention pooling, where the final representation is computed as a weighted sum of patch tokens:

where:

• $\text{output}$: final aggregated vision representation fed to the language model
• $\alpha_i$: learned attention weight for patch $i$, computed via multi-head attention with a learnable query vector (indicates importance of each patch)
• $v_i$: embedding vector for the $i$-th image patch
• $\sum_{i}$: weighted sum over all patches in the image

This formulation allows dynamic, content-dependent aggregation where the model learns which image regions to emphasize based on their relevance. Unlike a fixed CLS token aggregation, which must compress all information into a single vector regardless of content, attention pooling can focus on task-relevant regions. For instance, the model might emphasize text regions when performing OCR, facial regions when identifying people, or object boundaries during fine-grained recognition. This dynamic selection allows the encoder to adapt its output based on the visual content. This provides more informative representations to the language model than a fixed aggregation strategy could achieve.

Beyond single-vector pooling, many modern VLMs pass the full sequence of patch tokens to the language model rather than pooling at all. This approach, used in architectures like LLaVA and PaLI-Gemma, allows the language model to perform its own implicit spatial reasoning by attending to different patches depending on the question asked. The language model can attend to the patch tokens where the relevant answer appears, rather than relying on the vision encoder to anticipate which regions matter. This design trades computational efficiency for flexibility: passing all 729 patches from a SigLIP-SO400M-384 encoder to a 7B language model creates a prefix of 729 tokens that the language model must attend to on every generation step.