Vision Encoders for VLMs › Image Resolution Handling

Resolution handling represents one of the most active and challenging areas of VLM encoder research. The quadratic memory complexity of self-attention with respect to sequence length makes high-resolution processing computationally expensive, yet many practical tasks, including document understanding, scientific figure analysis, and fine-grained recognition, require pixel-level detail that only high resolution can provide.

For a ViT with patch size $p$ processing an image of height $h$ and width $w$, the sequence length is determined by:

$$N = \left\lceil \frac{h}{p} \right\rceil \times \left\lceil \frac{w}{p} \right\rceil$$

where:

• $N$: total number of patches (sequence length) fed to the transformer
• $h$: image height in pixels
• $w$: image width in pixels
• $p$: patch size (square patches assumed, so $p \times p$ pixels per patch)
• $\left\lceil \cdot \right\rceil$: ceiling function so partial patches are included

This formula demonstrates how image resolution translates directly to sequence length. Doubling both height and width quadruples $N$, which quadruples the attention memory since self-attention complexity scales with $O(N^2 \cdot d)$, where $d$ is the embedding dimension. Doubling image resolution while keeping patch size constant increases the number of patches by a factor of four, resulting in a sixteen-fold increase in attention computation.

Consider practical examples:

• 224x224 with 14x14 patches: 256 patches, manageable for most hardware
• 448x448 with 14x14 patches: 1,024 patches, requiring 4x the memory of 224px
• 896x896 with 14x14 patches: 4,096 patches, requiring 16x the memory of 224px

This quadratic scaling means that moving to higher resolutions quickly exhausts GPU memory, even with modern hardware, creating a basic tension between the need for fine detail and computational feasibility.

The memory cost compounds further in the context of a full VLM. The vision encoder processes the image once to produce patch tokens. These tokens then become part of the language model's context, and the language model must attend to all of them on every generation step during inference. A 7B parameter language model generating a response to a question about a high-resolution document image at 896x896 with 14x14 patches must attend to 4,096 vision tokens on top of the text tokens for every step of generation. On typical hardware, this quickly saturates the available memory budget for KV cache entries, severely limiting batch sizes or maximum output lengths.

Rather than resizing all images to a fixed square and distorting aspect ratios, modern VLM encoders often support native resolution processing, which maintains the original image proportions:

1. Dynamic padding: Images maintain their aspect ratio, with shorter sides padded with special padding tokens or zeros to match the required patch grid alignment. This prevents distortion but requires handling variable sequence lengths.

2. Variable sequence lengths: The encoder processes different image sizes with different sequence lengths, requiring the VLM to handle variable-length visual inputs. This is often managed through position-aware attention masking that indicates which tokens are real content versus padding.

3. Aspect ratio bucketing: During training, images are grouped into buckets with similar aspect ratios, such as 1:1 for portraits, 4:3 for standard photography, or 16:9 for widescreen images. This minimizes padding waste by so images in the same batch have consistent dimensions, then processes each bucket separately.

Maintaining native resolution and aspect ratio matters more for some content types than others. A photograph of a person's face looks natural when resized to a square because the face is roughly symmetric. A panoramic photograph distorted to a square might still be recognizable for scene understanding. But a document or spreadsheet distorted to a square becomes much harder to read: column alignments shift, text aspect ratios change, and tabular structure becomes ambiguous. For VLMs intended to handle business documents, invoices, or structured reports, native resolution processing is a correctness requirement rather than a quality enhancement.

When single-image processing hits memory limits, tiling strategies, also known as patch merging or sliding window approaches, divide high-resolution images into smaller, manageable crops:

• Non-overlapping tiles: Split the image into independent regions that are processed separately. While this approach is memory-efficient, it loses cross-tile context and may cut objects at tile boundaries, potentially missing relationships between objects in different tiles or fragmenting large objects.

• Overlapping tiles with global context: Process the full image at reduced resolution to capture global context, alongside high-resolution crops for local detail. The VLM learns to fuse these representations, combining the spatial precision of high-resolution crops with the semantic coherence of the global view. However, this approach doubles encoding cost since each image must be processed twice.

• Multi-scale feature pyramids: Process the image at multiple resolutions simultaneously, combining features from different scales before feeding to the language model. This resembles the hierarchical approaches in Sliding Window Attention but applies them to vision resolution, letting the model to attend to both fine details and global structure without prohibitive memory costs.

The overlapping tiles with global context strategy, used in models like LLaVA-HR and InternVL, has become particularly popular. It works by first processing the image at a standard low resolution, such as 224x224 or 336x336, to obtain a global understanding of the scene. Then the image is divided into a 2x2 or 3x3 grid of high-resolution tiles, each processed independently. The global representation provides context that prevents the model from misinterpreting tile-level details, while the tile representations provide the fine-grained spatial information needed for tasks like reading small text. A model asked whether a chart shows an upward or downward trend uses the global view to locate the chart axes and overall shape, then the high-resolution tiles to read the precise numerical labels on those axes.

Just as Position Interpolation extends language models to longer contexts beyond their training length, vision encoders can extend to higher resolutions through position encoding interpolation.

If a model was trained on 224x224 images, creating a 14x14 grid of 16x16 patches, extending to 448x448 resolution produces a 28x28 grid. This requires interpolating the learned position embeddings from the original 14x14 grid to the new 28x28 grid. Linear or bicubic interpolation works reasonably well for this expansion, though fine-tuning on higher resolution images significantly improves performance by letting the model to adapt to the new spatial frequencies and details available at higher resolution.

The interpolation quality depends significantly on how the position embeddings were originally learned. Embeddings trained with a smooth supervision signal, such as by minimizing reconstruction loss on position-perturbed images, interpolate more gracefully than embeddings learned purely from downstream task gradients. This is analogous to the observation in language models that position encodings trained with NTK-aware scaling interpolate more smoothly than those trained without attention to extrapolation properties, as we discussed in NTK-Aware Scaling.