Vision Transformer › The Inductive Bias Problem

The central trade-off of Vision Transformers is that they lack the visual inductive biases that make CNNs sample-efficient. To understand why this matters, it helps to examine exactly what these biases are and how they are embedded into the CNN architecture.

A CNN exploits two key architectural priors through the structure of convolution:

Locality: Nearby pixels form coherent patterns such as edges and textures alongside local color gradients. CNNs assume that meaningful features can be detected within small spatial neighborhoods. A $3 \times 3$ convolutional filter explicitly processes only a $3 \times 3$ region of the input at each position, ignoring all other regions. This forced locality constraint is a strong prior: it tells the network that relevant visual features are local. This is approximately true for low-level vision (edge detectors, blob detectors) and approximately true for mid-level features (corners, junctions) but less true for high-level semantics (object category), where global context often matters.

Translation equivariance: The same visual features should be detected regardless of their absolute position in the image. CNNs implement this through weight sharing: the same filter weights are applied at every spatial location. A horizontal edge detector learned for the top-left region automatically works for the bottom-right region, because the same weights slide across the entire image. This prior directly addresses the fact that the visual world is shift-invariant: a cat in the corner of an image looks the same as a cat in the center.

ViT has no such architectural assumptions. It must learn from data alone that adjacent patches correspond to adjacent image regions, and that visual features that appear at one spatial location also appear at others. Without weight sharing, there is no mechanism to enforce that what the model learns about patch 1 (top-left) applies to patch 100 (lower-right). The same cat, appearing at different positions, would produce different patch vectors, and the model must learn to handle this variation through exposure to many examples at many positions.

This is why the original ViT paper found that when pre-trained on ImageNet-21k (14 million images), ViT matched or exceeded ResNet performance on downstream tasks, but when trained only on ImageNet-1k (1.3 million images) from scratch, ResNets significantly outperformed ViT. The transformer architecture, without built-in visual priors, requires far more data to independently discover the locality and translation invariance that CNNs receive for free by architectural design.

The situation is analogous to learning grammar from scratch versus learning it from a teaching grammar. A child acquiring language receives structural guidance from the humans around them that constrains the hypothesis space. A model learning from raw text must infer grammatical structure from distributional statistics alone. Given enough data, both approaches produce equivalent linguistic competence, but the data-guided approach requires far more exposure to raw data to compensate for the absent structural guidance.

The visualization makes the contrast concrete. In a CNN, early layers have tiny receptive fields (the small inner circle) and global context only emerges after many stacked layers. In ViT, every patch can directly reach every other patch in the very first layer, with no sequential accumulation of context required. This global connectivity is simultaneously ViT's greatest architectural strength and its greatest data challenge.