Relative Position Encoding › Relative Position Encoding

Previous chapters explored absolute position encoding: each position receives a fixed representation based on its index in the sequence. Whether through sinusoidal functions or learned embeddings, the model learns that "position 5 means something specific." But language doesn't work this way. When you read "the cat sat on the mat," the relationship between "cat" and "sat" depends on their relative distance (one word apart), not on whether they appear at positions 2 and 3 versus positions 102 and 103. The same grammatical relationship holds whether the sentence opens a document or appears buried on page fifty.

Relative position encoding shifts the focus from "where am I?" to "how far apart are we?" This distinction sounds subtle, but it changes how models generalize to new sequences and new lengths. A model trained on short sequences may encounter position pairs during inference that it has never seen during training. With absolute encodings, this creates a generalization gap. With relative encodings, the model only needs to have seen a given distance during training, and that knowledge applies everywhere that same distance appears.

This chapter explores why that distinction matters, how to formulate relative attention mathematically, and how Shaw et al.'s (2018) influential approach brought relative positions into transformer self-attention in a practical, learnable way. The Shaw et al. paper is one of the most widely cited works in positional encoding because it introduced a clean, trainable formulation that could be dropped into existing transformer architectures with minimal modification.

Understanding relative position encoding also sets up important intuitions for the chapters ahead. Rotary Position Embeddings (RoPE) and ALiBi, two methods that power many of today's largest language models, both take the "encode distance, not index" philosophy and push it further with different mathematical formulations. By working through Shaw et al.'s approach first, you'll see both the core insight and the limitations that motivated those later innovations.

The material builds naturally on what you've already learned about self-attention and absolute position encodings. If you're comfortable with the query-key-value attention mechanism and understand why position information matters at all, you have everything you need to follow the derivation and implementation in this chapter.