Vision Transformer › Vision Transformer

For years, computer vision and natural language processing followed parallel but distinct trajectories, each developing specialized architectures suited to their respective modalities. While NLP underwent a revolutionary transformation with the transformer architecture, computer vision remained dominated by convolutional neural networks (CNNs). Architectures including ResNet, DenseNet, EfficientNet achieved impressive results by exploiting the inductive biases inherent in images: locality, which assumes that nearby pixels matter more than distant ones, and translation equivariance, which ensures that a cat remains a cat regardless of where it appears in the frame. These architectural priors allowed CNNs to learn efficiently from limited data, as the network structure itself encoded fundamental assumptions about how visual information is organized. Yet these same inductive biases that made CNNs sample-efficient also constrained their ability to model long-range dependencies and to scale gracefully with increasing amounts of data.

The Vision Transformer (ViT), introduced by Dosovitskiy et al. in 2020, posed a counterintuitive question: What if we set aside the convolutional inductive biases entirely and treat an image exactly like we treat text in a transformer? Instead of sliding convolutional filters across pixels to detect local features hierarchically, we could slice the image into uniform patches, linearly embed each patch into a vector representation, and process the resulting sequence with standard self-attention mechanisms. This approach seemed almost naive at first glance. CNNs had been engineered over decades to exploit the structure of images; removing those carefully designed priors should theoretically hurt performance, especially when working with limited training data where architectural constraints provide necessary regularization.

Yet when scaled appropriately with sufficient data and compute, ViT matched the performance of state-of-the-art CNNs and then surpassed them across numerous benchmarks. More importantly, ViT established a unified architectural foundation that enabled multimodal models to emerge. By representing both images and text as sequences of embeddings processed by identical transformer blocks, we can build systems that reason across modalities natively, using the same computational primitives for vision and language. As we discussed in Part XIII: Self-Attention, the transformer architecture's power derives from its ability to model arbitrary dependencies between sequence positions, regardless of distance. ViT demonstrated that this power extends naturally beyond discrete text tokens to continuous visual data, challenging the assumption that different modalities require fundamentally different architectural approaches.

Understanding ViT requires appreciating it as a conceptual shift rather than a technical curiosity. CNNs encode prior beliefs about images into their very structure. ViT instead asks: can we give the model enough data to discover those patterns on its own? The answer turns out to be yes, provided the dataset is large enough. This insight changes how we think about architecture design in deep learning more broadly: the more data you have, the less you need to bake prior knowledge into the architecture, and the more you can rely on the model to learn structure from scratch.

In this chapter, we explore in detail how ViT converts images into sequences through the patching and embedding process, examine the specific architectural details that differ from standard NLP transformers, and understand why ViT requires specific pre-training strategies to overcome its lack of built-in visual inductive biases. We trace the flow from raw pixels to classification logits, understanding both the mechanics and the theoretical underpinnings of this paradigm shift in computer vision.