Vision Transformer › Scaling Properties

Michael BrenndoerferFebruary 7, 20263 min read
Language AI HandbookAll chapters
Part 50 of 67Multimodal Models
Chapter 381Vision Transformer › Scaling Properties
Reading section 8 of 14 · Scaling Properties · 3 minSection 8 of 14 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 381 of 513

Section 8 of 14 · 3 min read

Vision Transformer › Scaling PropertiesOne section · Sections in this chapter8 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

ViT exhibits fundamentally different scaling characteristics compared to CNNs. While ResNet performance improves predictably but with diminishing returns as model size increases, ViT demonstrates steeper scaling curves when data is sufficient. With large datasets, larger ViT models continue to improve significantly, while CNN performance plateaus at a lower ceiling. This aligns with the scaling laws we explored in Part XXIII: Scaling Laws: transformer-based models follow power-law improvement with scale when data is abundant.

The original ViT paper introduced a family of standardized model sizes using naming conventions borrowed from BERT:

  • ViT-Tiny: embed_dim=192, depth=12, num_heads=3 (~5.7M parameters)
  • ViT-Small: embed_dim=384, depth=12, num_heads=6 (~22M parameters)
  • ViT-Base (ViT-B/16): embed_dim=768, depth=12, num_heads=12 (~86M parameters)
  • ViT-Large (ViT-L/16): embed_dim=1024, depth=24, num_heads=16 (~307M parameters)
  • ViT-Huge (ViT-H/14): embed_dim=1280, depth=32, num_heads=16, patch_size=14 (~632M parameters)

The notation ViT-B/16 means Base model with 16×1616 \times 16 patches; ViT-H/14 uses 14×1414 \times 14 patches instead. Moving to smaller patches for the Huge model increases the sequence length and allows the model's greater capacity to capture finer spatial detail.

Out[6]:
Visualization
Line chart on a log x-axis comparing Top-1 accuracy of ResNet and ViT across dataset sizes from 100k to 300M images, showing a crossover where ViT surpasses ResNet at large data scales.
Simulated performance scaling with dataset size for ViT versus ResNet. Vision Transformers demonstrate steeper scaling curves, surpassing ResNet when trained on sufficiently large datasets despite inferior performance at smaller data scales due to their lack of visual inductive biases.

The crossover point in the scaling curve reflects the core trade-off: CNNs start with strong inductive priors that provide a head start in the low-data regime, but these same priors impose a ceiling on what the model can learn. ViT starts with no visual priors, requiring more data to learn the basics, but its more flexible representational capacity allows it to keep improving at larger scales.