Relative Position Encoding › Summary

Relative position encoding shifts the focus from "where am I?" to "how far apart are we?" By encoding the offset between positions rather than absolute indices, models learn patterns that generalize across all positions at the same distance.

Key takeaways from this chapter:

• Absolute positions limit generalization: A model trained on short sequences may never see position pairs that appear in longer sequences. Relative encoding ensures that distance patterns transfer regardless of absolute position. Nearly 75% of position pairs in a 2x-length test sequence are unseen during training on the shorter length.

• Shaw et al. formulation: Modify attention scores by adding $\mathbf{q}_i \cdot \mathbf{a}^K_{j-i}$ to the content term. This separates semantic compatibility from distance-based compatibility. The position term is a dot product between the query and the relative position embedding, allowing different query types to respond differently to the same distance.

• Clipping bounds memory: Relative positions beyond distance $k$ share the same embedding. This limits the number of parameters to $2k + 1$ regardless of sequence length, and also prevents distant embeddings from being undertrained due to infrequent occurrence.

• Two components to modify: Shaw et al. add relative embeddings to both keys (affecting which positions get attention) and values (affecting what content flows). Both contribute to the learned distance patterns, though many implementations omit the value-side for efficiency.

• Diagonal structure in attention: Position biases form a Toeplitz-like pattern where each anti-diagonal corresponds to a fixed relative distance. This structure enables efficient computation by precomputing all query-offset dot products in a single matrix multiply.

• Trade-off between complexity and generalization: Relative position encoding adds computational overhead (roughly 5-10% at training time) but provides better generalization to unseen position pairs and longer sequences. The overhead grows with sequence length, which has motivated subsequent methods designed for long-context settings.

In the next chapter, we'll explore Rotary Position Embeddings (RoPE), a method that encodes relative positions through rotation rather than additive biases. RoPE achieves similar generalization benefits with a more elegant mathematical formulation that integrates naturally with the attention computation, and it has become the dominant approach in large language models such as LLaMA and Mistral, along with GPT-NeoX.