Vision Transformer › Position Embeddings in Vision

Recall from Part XIV: Positional Encoding that transformers process sequences in parallel and have no inherent notion of token order or sequence position. In NLP, we explored sinusoidal encodings, learned embeddings, and relative position representations to inject ordering information. For vision, the question is more complex: images have inherently 2D spatial structure, but our patching operation has flattened them into a 1D sequence. How do we encode 2D spatial relationships when the sequence is 1D?

The original ViT used standard 1D learned position embeddings, treating the sequence of patches exactly like a sequence of word tokens in a language model. Each position index $i \in \{0, 1, \ldots, N\}$ has an associated learnable vector $\mathbf{e}_i \in \mathbb{R}^D$ that is added to the token at that position. These position embeddings are initialized randomly and trained alongside the rest of the model. Remarkably, this simple approach works well in practice despite its apparent disregard for the 2D nature of images.

Why does 1D learned position encoding work for 2D images? The key insight is that the 2D spatial arrangement of patches maps directly to a specific 1D ordering in row-major format. Patch $(r, c)$ always appears at position $i = r \cdot (W/P) + c$. The model can learn position embeddings such that patches at similar 2D positions receive similar position embeddings. If patch at position $i = 14$ (row 1, column 0) and patch at position $i = 1$ (row 0, column 1) both have position embeddings that share useful features, the model can learn spatial adjacency implicitly. Research has confirmed that after training, the learned 1D position embeddings indeed exhibit 2D spatial structure: when you compute cosine similarities between position embeddings, spatially adjacent patches have more similar embeddings than distant ones.

Alternative position encoding strategies that have been explored in ViT variants include:

• 2D position embeddings: Use separate row and column embeddings, summed to produce a composite spatial representation:

  $$\mathbf{E}_{\text{pos}}^{(r,c)} = \mathbf{E}_{\text{row}}^{(r)} + \mathbf{E}_{\text{col}}^{(c)}$$

  where $\mathbf{E}_{\text{row}}^{(r)} \in \mathbb{R}^{D}$ is the learned embedding for row $r$ and $\mathbf{E}_{\text{col}}^{(c)} \in \mathbb{R}^{D}$ is the learned embedding for column $c$. This approach explicitly factors spatial position into orthogonal row and column components, which may make it easier for the model to generalize to images of different resolutions.

• Relative position encodings: Instead of encoding absolute positions, encode the relative offset between any two patches. When computing attention between patch $i$ and patch $j$, a bias term is added that depends on $\Delta r = r_i - r_j$ and $\Delta c = c_i - c_j$. This approach generalizes naturally to images of different sizes and can better capture local spatial relationships.

• 2D sinusoidal encodings: Extend the sinusoidal position encoding from Sinusoidal Position Encoding to two dimensions by creating sinusoidal patterns across both height and width axes independently, then concatenating or summing them.

Empirically, the differences between these approaches are relatively small for classification tasks on fixed-resolution images, suggesting that the transformer is remarkably effective at learning spatial relationships from 1D position embeddings given sufficient training. For tasks requiring fine-grained spatial reasoning or adaptation to variable resolutions, 2D or relative position encodings offer modest improvements. The flexibility of attention-based position encoding, compared to the fixed positional assumptions embedded in CNN architectures, is itself an advantage of the ViT approach.

The block-diagonal structure in the heatmap reveals a key property: patches from the same row of the spatial grid (separated by 7 positions in the 1D sequence) show higher similarity to each other than patches from distant rows. This demonstrates that learned 1D position embeddings spontaneously develop 2D spatial awareness during training, even without an explicit 2D structure in the encoding scheme.