Relative Position Encoding › Limitations and Practical Considerations

Relative position encoding solves the generalization problem but introduces its own challenges. Understanding these limitations is important both for choosing between absolute and relative encoding in a new project and for anticipating where Shaw et al.'s approach will fall short.

The most significant limitation is computational complexity. Standard absolute position encoding adds to embeddings once at the input, with no additional cost during attention. Relative position encoding modifies every attention score computation. For a sequence of length $n$, we compute $n^2$ relative position biases per attention layer. With multiple layers and attention heads, this overhead accumulates. Shaw et al. note that their approach adds roughly 5-10% to training time, which is acceptable for most applications. However, for very long sequences (4096+ tokens), even a 10% overhead becomes significant because attention is already the bottleneck, and any additional computation in the $O(n^2)$ inner loop compounds quickly.

Memory usage also increases. We store $(2k+1)$ relative position embeddings per dimension, for both keys and values. With $d_k = d_v = 64$ and $k = 64$, this adds about 16K parameters per attention head. This is modest compared to the projection matrices, but the position bias matrix computed during the forward pass has shape $(n, n)$, which for $n = 4096$ requires 64 MB per layer just for the bias storage (in float32). For very long sequences, this memory footprint rivals the attention weight matrix itself.

The clipping distance $k$ introduces a hyperparameter choice with no universally correct answer. Too small, and the model loses fine-grained distance information for medium-range dependencies. Too large, and distant embeddings have insufficient training signal, particularly early in training when most gradients flow through frequent short-distance pairs. Shaw et al. used $k = 16$ in their original experiments on the WMT translation benchmark, but this value is too small for tasks involving long-range dependencies in English prose. In practice, $k$ is often set to match the typical context length during training, such as 64 or 128 for sentence-level tasks.

A subtler limitation is that Shaw et al.'s formulation encodes position at a single point in the transformer architecture: the attention score computation. This means position information can only influence the model through the routing decision (which tokens get attention) and the value aggregation (what information flows). It cannot influence the feedforward sublayers, layer norm operations, or any other component. More recent approaches like RoPE embed position information in a way that is more deeply integrated into the computation, allowing it to influence behavior throughout the network.

Finally, Shaw et al.'s relative position embeddings are fully learned, which means they depend on having sufficient training data to cover the full range of distance patterns the model needs to handle. For tasks with very limited data, the distant-position embeddings may remain undertrained throughout fine-tuning, creating a different kind of generalization problem than the one we set out to solve. In such cases, a fixed relative position scheme (like the sinusoidal approach used in Transformer-XL) may generalize better.

Despite these limitations, relative position encoding's impact on transformer research has been substantial. It demonstrated that position information can be injected at the attention level rather than the embedding level, opening the door to more sophisticated schemes like Rotary Position Embeddings and ALiBi that we'll explore in subsequent chapters. Every major language model architecture developed since 2019 has incorporated some form of relative position encoding, and the foundational ideas in this chapter appear in all of them.