Position Problem › Absolute vs. Relative Position

Each approach has significant trade-offs. Absolute encoding is simpler to implement: just add a position vector to each token embedding before feeding it to the model. The attention mechanism then sees combined content-position vectors and can, with sufficient capacity, learn to extract relative positional information by computing differences between absolute positions. The original Transformer and later models such as BERT and GPT all use absolute positional encoding.

The limitation of absolute encoding is that it requires the model to learn the mapping from absolute to relative positions implicitly. If the model sees "position 3 attends to position 1" during training, it must generalize to "position 103 attends to position 101" at inference. These are the same relative offset (+2) expressed as different absolute positions. With enough data and capacity, models can learn this generalization, but it is not guaranteed, and it may require more training examples than relative encoding.

Relative encoding directly captures these distance-based relationships. Instead of embedding absolute positions, the model embeds the distances $i - j$ between query position $i$ and key position $j$. This representation is translation-invariant by construction: a distance of +2 means the same thing regardless of whether we are at positions 1 and 3 or positions 101 and 103. The model does not need to generalize across absolute positions.

The drawback of relative encoding is architectural complexity. Absolute positional vectors can be added to token embeddings once at the input layer, and the rest of the model is unchanged. Relative positions depend on pairs of positions, not individual positions, so they cannot simply be added to the input. The attention score between positions $i$ and $j$ must explicitly incorporate information about the distance $i - j$. This requires modifying the attention computation itself, rather than changing only the input preprocessing. The result is a more expressive model that is also harder to implement efficiently.

Modern architectures increasingly favor relative or hybrid approaches. RoPE (Rotary Position Embedding) cleverly encodes relative positions through rotation operations that integrate naturally with the standard attention dot product. ALiBi (Attention with Linear Biases) adds a simple learned bias to attention scores based on distance, achieving relative encoding with minimal architectural change. We will explore both approaches in dedicated chapters.

The preference for relative encoding in recent architectures reflects a linguistic observation about local relationships. Most syntactic and semantic relationships in language are local and relative, not global and absolute. A subject typically appears within a few words of its verb, and this proximity holds whether the subject-verb pair appears at the beginning of a sentence or in the middle of a long document. A determiner ("the," "a") always immediately precedes the noun phrase it introduces, and this is a relative relationship independent of absolute position. Anaphoric pronouns refer back to noun phrases that appeared somewhere earlier in the discourse, a backward-pointing relative relationship.

If positional representations encode relative distances well, the model can more easily learn these universal syntactic generalizations. If they encode only absolute positions, the model must learn to apply each generalization at every possible absolute location, multiplying the number of patterns it needs to internalize.

This does not mean absolute encoding is useless. Document-level reasoning, reference tracking across very long spans, and certain ordering tasks may benefit from knowing absolute position. A model that knows it is processing the first sentence of a document versus the twentieth sentence might behave differently. Hybrid approaches that encode both absolute and relative information are an active area of research.