Vision Transformer › Image Patching: From Pixels to Sequences

The fundamental challenge in applying transformers to images is a dimensional mismatch between what the architecture expects and how visual data is structured. A standard transformer encoder expects input as a sequence of vectors $\mathbf{X} \in \mathbb{R}^{N \times D}$, where $N$ is the sequence length and $D$ is the embedding dimension. A typical image, however, arrives as a 3D tensor $\mathbf{I} \in \mathbb{R}^{H \times W \times C}$ representing height and width plus color channels. This structural difference poses an immediate problem: transformers process sequences, while images are inherently two-dimensional grids with channel information.

The solution ViT adopts is elegant in its simplicity: reshape the image into a sequence by dividing it into a regular grid of fixed-size square patches. Each patch is flattened into a single vector by concatenating all pixel values across spatial locations and color channels. These flattened vectors then serve as the input sequence to the transformer, with each patch playing the role that a word token plays in a language model.

Consider a standard input image of size $224 \times 224$ pixels with 3 color channels (RGB). If we choose a patch size of $P = 16$ pixels per side, the patching proceeds as follows. First, we divide the image into a $14 \times 14$ grid of non-overlapping patches, since $224 / 16 = 14$ patches fit along each spatial dimension. The total number of patches is:

Each individual patch is a $16 \times 16 \times 3$ sub-image containing $16 \times 16 \times 3 = 768$ values. We flatten this 3D sub-image into a 1D vector by concatenating all pixel values in row-major order across height and width for every channel. More precisely, for the patch at grid position $(r, c)$ where $r, c \in \{0, 1, \ldots, 13\}$, we extract:

where:

• $\mathbf{x}^{(r,c)} \in \mathbb{R}^{P^2 \cdot C}$: the flattened patch vector at grid row $r$, grid column $c$
• $\mathbf{I}[r \cdot P : (r+1) \cdot P, c \cdot P : (c+1) \cdot P, :]$: the slice of the image tensor corresponding to the patch at position $(r, c)$, with all channels included
• $P$: the patch size (number of pixels per side, e.g., 16)
• $C$: the number of color channels (3 for RGB)
• $\text{flatten}(\cdot)$: the operation that reshapes a $P \times P \times C$ tensor into a vector of length $P^2 \cdot C$

We then flatten the 2D grid of patches into a 1D sequence by linearizing the grid positions. Conventionally, this is done in row-major order: patches in the first row of the grid appear first in the sequence, followed by patches in the second row, and so on. Patch $(r, c)$ gets sequence index $i = r \times (W/P) + c$. We now have a sequence $[\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]$ of $N = 196$ vectors, each of dimension 768. This is a sequence length and dimensionality comparable to what standard language transformers process, making the same architecture applicable with minimal modification.

It is worth pausing to appreciate what has been lost and what has been retained in this transformation. We have retained the pixel values of every region of the image; no information has been discarded or aggregated. What we have lost is the explicit 2D spatial structure. The sequence representation no longer encodes the fact that patch 1 is adjacent to patch 2 horizontally, or that patch 15 lies directly below patch 1. The transformer receives an unordered bag of patch vectors with positional indices, and it must learn from those indices and from training data which patches are spatially related. This is the inductive bias gap that we will discuss in depth later.

The patching approach also imposes an implicit tokenization granularity. A word token in NLP typically carries semantic content: the word "cat" has a meaning that the model can use directly. A $16 \times 16$ patch of pixels, by contrast, often contains no clear semantic unit. It might span the boundary between a cat's ear and the background, or contain only a uniform textured region with no object at all. This means patch-level representations start at a much lower semantic level than word tokens, requiring the model to learn compositional semantics through attention across patches, rather than beginning with semantically meaningful units. This is another reason why ViT needs more data: it must learn both low-level visual features and their high-level composition, whereas NLP models begin with tokens that already carry semantic weight encoded by the vocabulary.