Vision Encoders for VLMs › Worked Example: Resolution and Memory
Language AI HandbookAll chapters Vision Encoders for VLMs › Worked Example: Resolution and MemoryOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us calculate the computational implications of different resolution choices for a VLM vision encoder to understand the practical constraints facing system designers.
Consider a ViT-L/14 encoder, which is the Large variant with 14x14 patches and 1024 hidden dimensions, processing images at various resolutions:
def calculate_vision_cost(resolution, patch_size, hidden_dim, batch_size=1):
"""
Calculate memory and compute requirements for ViT processing.
Args:
resolution: Tuple of (height, width)
patch_size: Size of square patches
hidden_dim: Hidden dimension of transformer
batch_size: Number of images in batch
"""
h, w = resolution
patches_h = (h + patch_size - 1) // patch_size # Ceiling division
patches_w = (w + patch_size - 1) // patch_size
num_patches = patches_h * patches_w
# Memory for patch embeddings (4 bytes per float32)
embedding_memory = (
batch_size * num_patches * hidden_dim * 4 / (1024**2)
) # MB
# Attention matrix memory (Q@K^T): batch * heads * patches * patches * 4 bytes
# Assume 16 heads for ViT-L
num_heads = 16
attention_memory = (
batch_size * num_heads * num_patches * num_patches * 4 / (1024**2)
) # MB
# Approximate FLOPs for attention: 2 * batch * patches^2 * hidden_dim
attention_flops = (
2 * batch_size * num_patches * num_patches * hidden_dim / 1e9
) # GFLOPs
return {
"patches": num_patches,
"sequence_length": num_patches + 1, # +1 for CLS token
"embedding_memory_mb": embedding_memory,
"attention_memory_mb": attention_memory,
"attention_gflops": attention_flops,
}
# Common configurations
configs = [
(224, 224, 14, 1024, "Standard (224px)"),
(336, 336, 14, 1024, "High-res (336px)"),
(448, 448, 14, 1024, "Very high-res (448px)"),
(224, 224, 16, 1024, "Large patches (16px)"),
(384, 384, 14, 768, "SigLIP Base (384px)"),
]Configuration Patches Emb Mem (MB) Attn Mem (MB) Attn GFLOPs -------------------------------------------------------------------------------- Standard (224px) 256 1.00 4.00 0.13 High-res (336px) 576 2.25 20.25 0.68 Very high-res (448px) 1024 4.00 64.00 2.15 Large patches (16px) 196 0.77 2.34 0.08 SigLIP Base (384px) 784 2.30 37.52 0.94
The table reveals why resolution scaling presents such a formidable challenge in VLM design. Moving from 224px to 448px increases attention memory by 16x, from approximately 3 MB to 50 MB per layer, and increases compute by 16x as well. For a 24-layer ViT-L, this represents the difference between fitting comfortably on a consumer GPU and requiring enterprise-grade hardware with substantial memory capacity.
Notice that increasing patch size from 14 to 16 reduces sequence length by 23%, from 256 patches to 196 patches. It offers a simple efficiency lever at the cost of spatial precision. This tradeoff between patch size and resolution represents a basic design decision in vision encoder architecture. The SigLIP Base configuration at 384x384 achieves a sequence length of 729 with a 768-dimensional embedding, compared to 1024 dimensions in the ViT-L configurations. The lower dimensionality helps offset the higher patch count. This shows how careful co-design of patch size and resolution alongside embedding dimension can achieve a better computational footprint than tuning any single parameter in isolation.