Relative Position Encoding

Previous chapters explored absolute position encoding: each position receives a fixed representation based on its index in the sequence. Whether through sinusoidal functions or learned embeddings, the model learns that "position 5 means something specific." But language doesn't work this way. When you read "the cat sat on the mat," the relationship between "cat" and "sat" depends on their relative distance (one word apart), not on whether they appear at positions 2 and 3 versus positions 102 and 103. The same grammatical relationship holds whether the sentence opens a document or appears buried on page fifty.

Relative position encoding shifts the focus from "where am I?" to "how far apart are we?" This distinction sounds subtle, but it changes how models generalize to new sequences and new lengths. A model trained on short sequences may encounter position pairs during inference that it has never seen during training. With absolute encodings, this creates a generalization gap. With relative encodings, the model only needs to have seen a given distance during training, and that knowledge applies everywhere that same distance appears.

This chapter explores why that distinction matters, how to formulate relative attention mathematically, and how Shaw et al.'s (2018) influential approach brought relative positions into transformer self-attention in a practical, learnable way. The Shaw et al. paper is one of the most widely cited works in positional encoding because it introduced a clean, trainable formulation that could be dropped into existing transformer architectures with minimal modification.

Understanding relative position encoding also sets up important intuitions for the chapters ahead. Rotary Position Embeddings (RoPE) and ALiBi, two methods that power many of today's largest language models, both take the "encode distance, not index" philosophy and push it further with different mathematical formulations. By working through Shaw et al.'s approach first, you'll see both the core insight and the limitations that motivated those later innovations.

The material builds naturally on what you've already learned about self-attention and absolute position encodings. If you're comfortable with the query-key-value attention mechanism and understand why position information matters at all, you have everything you need to follow the derivation and implementation in this chapter.

Consider the sentence "The cat that I saw yesterday sat on the mat." The verb "sat" needs to find its subject "cat." With absolute position encoding, the model must learn that "position 5 attending to position 2" captures a subject-verb relationship. But if we move the same phrase later in the document, it becomes "position 105 attending to position 102." The model must learn these patterns for every possible absolute position pair, treating what is essentially the same linguistic relationship as hundreds of distinct cases.

Relative position encoding solves this by focusing on the offset between positions. The distance from position 5 to position 2 is $-3$ (looking backward three positions). The distance from position 105 to position 102 is also $-3$. By encoding this offset rather than absolute positions, the model learns a single pattern for "three positions back" that generalizes across the entire sequence.

This shift has practical consequences that go beyond test-set metrics. Language has many distance-dependent patterns: adjectives typically appear one position before their nouns, determiners precede noun phrases by a few positions, and verbs often follow their subjects within a local window. These patterns are not anchored to absolute positions; they describe structural relationships that can appear anywhere in a document. Relative encoding captures these patterns directly, letting a single learned representation cover all occurrences of a given structural relationship regardless of where that relationship appears in the input.

The generalization benefit becomes especially important for long-context applications. A model trained on sequences of 512 tokens and then applied to sequences of 2048 tokens will encounter absolute positions it has never seen. Absolute encodings have no principled way to handle this: the embeddings for positions 513 through 2048 were never trained. Relative encodings, by contrast, are anchored to distances rather than indices. As long as the model has seen sequences where tokens are $k$ positions apart, it can apply that knowledge at any absolute location where that same distance appears.

Nearly 75% of position pairs at test time were never seen during training. A model relying on absolute positions must somehow generalize to these unseen combinations, either by interpolating from nearby positions or by pattern-matching to semantically similar content. Neither approach is principled. With relative encoding, every distance seen during training transfers to all positions at that distance, turning a difficult extrapolation problem into a simple reuse of already-learned representations.

Recall the standard self-attention formula. Given queries $\mathbf{Q}$, keys $\mathbf{K}$, and values $\mathbf{V}$, we compute:

where:

• $\mathbf{Q} \in \mathbb{R}^{n \times d_k}$: query matrix containing all query vectors
• $\mathbf{K} \in \mathbb{R}^{n \times d_k}$: key matrix containing all key vectors
• $\mathbf{V} \in \mathbb{R}^{n \times d_v}$: value matrix containing all value vectors
• $d_k$: dimension of queries and keys
• $\sqrt{d_k}$: scaling factor to prevent dot products from growing too large
• $n$: sequence length

With absolute position encoding, each token's embedding includes position information before the QKV projections. Position is baked into the input: $\mathbf{x}_i = \mathbf{e}_i + \mathbf{p}_i$, where $\mathbf{e}_i$ is the token embedding and $\mathbf{p}_i$ is the position encoding. The attention scores then implicitly depend on absolute positions through the projected queries and keys. The model sees position information, but only indirectly: it must disentangle "what is this token?" from "where is this token?" inside the projection matrices themselves.

Relative position encoding takes a different approach: inject position information directly into the attention computation. Instead of adding position to the input, we modify how attention scores are calculated to explicitly account for the offset $j - i$ between positions $i$ and $j$. This is a cleaner separation of concerns. Content similarity handles semantic matching; a separate position-dependent term handles distance-based preferences. The model does not need to learn to disentangle them because they are never mixed in the first place.

The core idea is to add a position-dependent term to the attention score. To give the model a sense of why tokens at certain distances should receive more or less attention, we write:

where:

• $\mathbf{q}_i$: the query vector at position $i$
• $\mathbf{k}_j$: the key vector at position $j$
• $\text{relative\_bias}(i, j)$: a term that depends only on the offset $j - i$

This formulation separates content matching ($\mathbf{q}_i \cdot \mathbf{k}_j$) from position matching (the relative bias). The model can learn that certain offsets are generally more or less important, regardless of absolute position. Notice that the relative bias only depends on the difference $j - i$, not on $i$ and $j$ individually. This is the key mathematical property that enables generalization: two position pairs with the same offset get the same positional treatment, no matter where they appear in the sequence.

The most influential formulation of relative positions in self-attention comes from Shaw et al. (2018). Their approach asks a fundamental question: if we want attention to be distance-aware, where exactly should we inject position information? Rather than modifying the input embeddings, Shaw et al. discovered that injecting relative positions directly into the attention computation gives the model finer control over how distance affects both which tokens get attention and what information flows.

To understand their approach, we need to build up three connected ideas: what we want the model to learn about distance, how to represent those distances as learnable parameters, and how to integrate those representations into the attention mechanism itself.

Consider what happens when position $i$ attends to position $j$ in standard self-attention. The model computes a score based on content similarity, then uses that score to weight how much of $j$'s information flows to $i$. But we want distance to influence both of these operations independently.

Think of it this way: a verb attending to possible subjects has two separate questions to answer. First, which position is the most likely subject based on both content and distance? Second, once the subject is found, how much of its information should flow into the verb's representation? Both questions depend on distance, but in different ways. The model might want to assign high attention to a subject that is two positions back, and to weight that subject's contribution with a "subject-attended-from-two-back" color that differs from "subject-attended-from-ten-back."

Shaw et al. address both needs by introducing two sets of learnable embeddings: one that modifies attention scores (controlling who gets attention), and one that modifies value aggregation (controlling what information flows). This separation gives the model independent control over "who to attend to" versus "what to gather."

1. Which positions get attention: A verb might prefer to attend to positions 1-3 positions back (where subjects typically appear), regardless of content similarity.

2. What information flows: When aggregating information, we might want "nearby context" to carry different weight than "distant context," even when attention weights are equal.

For each possible offset $r = j - i$ between positions, Shaw et al. define learnable vectors that the model adjusts during training just like any other parameter:

• $\mathbf{a}^K_r \in \mathbb{R}^{d_k}$: a relative position embedding that influences attention scores (key-side)
• $\mathbf{a}^V_r \in \mathbb{R}^{d_v}$: a relative position embedding that modifies aggregated values (value-side)

where:

• $r = j - i$: the relative offset (positive means looking forward, negative means looking backward)
• $d_k$: dimension of query and key vectors
• $d_v$: dimension of value vectors

These embeddings are learned during training, just like the QKV projection matrices. The critical insight is that the number of distinct offsets is bounded by the sequence length (from $-(n-1)$ to $+(n-1)$), not by the number of position pairs (which is $n^2$). This means the model learns a single pattern for "three positions back" that applies universally, rather than learning separate patterns for every pair of absolute positions. The parameter count scales linearly with the clipping distance $k$, not quadratically with sequence length.

The key-side embeddings ($\mathbf{a}^K$) and value-side embeddings ($\mathbf{a}^V$) are separate parameter matrices, not tied to each other. This gives the model full flexibility: the same distance can produce different effects on the routing decision (score computation) versus the aggregation decision (value weighting).

Let's trace through how relative positions enter the attention computation. In standard self-attention, the score measuring how much position $i$ should attend to position $j$ captures only content similarity:

where:

• $e_{ij}$: the raw attention score from query position $i$ to key position $j$
• $\mathbf{q}_i \in \mathbb{R}^{d_k}$: the query vector at position $i$
• $\mathbf{k}_j \in \mathbb{R}^{d_k}$: the key vector at position $j$
• $d_k$: the dimension of query and key vectors

This score depends only on the content at each position. Two identical words at different positions produce identical scores, regardless of their distance. The model cannot distinguish "I saw a cat (right here, adjacent)" from "I saw a cat (twenty words ago)."

Shaw et al. modify this by adding the relative position embedding to the key before computing the dot product:

Why add to the key rather than the query? The key represents "what this position offers," so adding position information to it means the model can learn that positions at certain distances offer something different, even if their content is identical. The query remains focused on "what I'm looking for" without distance context; the key communicates "here is what I have, adjusted for how far away I am from the query."

Expanding the dot product reveals the two-component structure:

where:

• $\mathbf{q}_i \cdot \mathbf{k}_j$: the content term, measuring semantic compatibility between positions
• $\mathbf{q}_i \cdot \mathbf{a}^K_{j-i}$: the position term, encoding how relative distance affects the score
• $\mathbf{a}^K_{j-i} \in \mathbb{R}^{d_k}$: the learned relative position embedding for offset $(j-i)$

The position term is a dot product between the query and the relative position embedding. This means different queries can respond differently to the same distance. A query learned to find subjects might have a high dot product with $\mathbf{a}^K_{-2}$ (two positions back), while a query learned to find objects might prefer $\mathbf{a}^K_{+1}$ (one position forward). The model learns these preferences end-to-end from training data, so the resulting position patterns reflect actual linguistic structure rather than any hand-designed prior.

Shaw et al. apply the same principle to value aggregation. After computing attention weights $\alpha_{ij} = \text{softmax}(e_{ij})$, standard attention aggregates values as a weighted sum. We want to add relative position information here too, so that the output at each position carries the weighted content of attended tokens and a record of where that content came from.

The modified value aggregation is:

where:

• $\mathbf{o}_i \in \mathbb{R}^{d_v}$: output vector at position $i$, now enriched with position-aware context
• $\alpha_{ij}$: attention weight from position $i$ to position $j$ (sums to 1 across $j$)
• $\mathbf{v}_j \in \mathbb{R}^{d_v}$: value vector at position $j$, carrying the content to aggregate
• $\mathbf{a}^V_{j-i} \in \mathbb{R}^{d_v}$: relative position embedding for values at offset $(j-i)$
• $n$: sequence length

This modification means the output at each position incorporates the weighted content from other positions and position-dependent information about where that content came from. Even if two positions have identical values and identical attention weights, their contributions differ based on their relative distances. Think of it as: "I got this information from a token that was three positions back, and that proximity matters for how I should interpret it."

The two sets of embeddings ($\mathbf{a}^K$ and $\mathbf{a}^V$) work together but serve distinct roles. The key embeddings affect which positions receive attention (the "routing" decision). The value embeddings affect what information flows once routing is decided (the "content" decision). This separation allows the model to learn, for example, that adjacent positions should receive high attention (key embedding effect) while also learning that adjacent context should contribute a specific "locality" signal to the output (value embedding effect).

In practice, many implementations omit the value-side embeddings ($\mathbf{a}^V$) for efficiency reasons. The key-side modification tends to deliver most of the benefit, and removing the value modification cuts the number of relative position parameters in half. The T5 model, for instance, uses only key-side biases. Shaw et al.'s original paper includes both, but their ablation experiments showed that omitting the value-side embeddings costs only a small amount of task performance.

The formulation above assumes we have a separate embedding for every possible relative position. But this creates a practical problem: for a sequence of length $n$, relative positions range from $-(n-1)$ to $(n-1)$, requiring $2n-1$ embeddings. For a sequence of length 512, that's 1023 different embeddings per dimension, for both keys and values.

This approach has two flaws. First, memory grows linearly with sequence length. Second, distant positions appear infrequently during training. In a 128-token sequence, only one position pair has offset $+127$ (position 0 to position 127), while thousands of pairs have offset $+1$. The embedding for offset $+127$ would be severely undertrained: the gradient updates for that embedding come from only one pair per sequence, while the embedding for offset $+1$ receives gradient contributions from 127 pairs. The result is that distant-position embeddings never become reliable.

Shaw et al. address both problems with a simple insight: fine-grained distance matters more locally than globally. The difference between "3 positions back" and "4 positions back" may be linguistically significant, as it could distinguish an adjective-noun relationship from a determiner-noun relationship. But the difference between "100 positions back" and "101 positions back" rarely carries meaning. Both are simply "far away." The model can usefully represent "very distant" as a single category without losing important information.

This motivates clipping: relative positions beyond a maximum distance $k$ are clamped to $\pm k$:

where:

• $j - i$: the raw relative position (offset from position $i$ to position $j$)
• $k$: the maximum relative position to distinguish (a hyperparameter)
• $\text{clip}(x, a, b)$: clamps $x$ to the range $[a, b]$, returning $a$ if $x < a$, $b$ if $x > b$, and $x$ otherwise

With clipping, we need only $2k + 1$ relative position embeddings regardless of sequence length. Position pairs further than $k$ apart share the same "far away" embedding, either $\mathbf{a}^K_{-k}$ for "far backward" or $\mathbf{a}^K_{+k}$ for "far forward."

The choice of $k$ reflects a trade-off between expressiveness and learnability. Typical values range from 16 to 128, depending on the expected locality of relevant patterns in the data. For sentence-level tasks where the longest dependencies span 10-20 tokens, $k = 16$ often suffices. For document-level tasks with paragraph-scale dependencies, $k = 64$ or $k = 128$ gives the model more distance resolution. In practice, the performance difference between $k = 32$ and $k = 64$ is often small, but the difference between $k = 4$ and $k = 16$ can be significant for tasks involving long-range grammatical dependencies.

Notice the diagonal band structure: each anti-diagonal contains the same relative position. The corners, where positions are far apart, are clipped to the boundary values $\pm k$. This Toeplitz-like structure is key to efficient implementation because it means position biases repeat in a predictable pattern that can be computed once and reused.

Let's walk through a concrete, minimal example to make the math tangible. Consider a four-token sequence with one-dimensional embeddings, maximum clipping distance $k = 2$, and very simple relative position embeddings. The goal is to show exactly where the relative bias comes from and how it modifies the attention scores.

Suppose we have queries $\mathbf{q}_1 = [1.0]$, $\mathbf{q}_2 = [0.5]$, $\mathbf{q}_3 = [0.8]$, $\mathbf{q}_4 = [0.3]$ and keys $\mathbf{k}_1 = [0.9]$, $\mathbf{k}_2 = [0.7]$, $\mathbf{k}_3 = [0.4]$, $\mathbf{k}_4 = [1.2]$. The key-side relative position embeddings (after clipping to $k = 2$) are:

• $\mathbf{a}^K_{-2} = [-0.3]$ (two positions back)
• $\mathbf{a}^K_{-1} = [+0.4]$ (one position back)
• $\mathbf{a}^K_{0} = [+0.1]$ (same position)
• $\mathbf{a}^K_{+1} = [-0.2]$ (one position forward)
• $\mathbf{a}^K_{+2} = [-0.5]$ (two positions forward)

Now let's compute the score from query position 1 to all key positions. With $d_k = 1$ so $\sqrt{d_k} = 1$:

For position pair $(i=1, j=1)$: offset is $0$, content term is $1.0 \times 0.9 = 0.9$, position term is $1.0 \times 0.1 = 0.1$. Combined score: $1.0$.

For position pair $(i=1, j=2)$: offset is $+1$, content term is $1.0 \times 0.7 = 0.7$, position term is $1.0 \times (-0.2) = -0.2$. Combined score: $0.5$.

For position pair $(i=1, j=3)$: offset is $+2$, content term is $1.0 \times 0.4 = 0.4$, position term is $1.0 \times (-0.5) = -0.5$. Combined score: $-0.1$.

For position pair $(i=1, j=4)$: offset is $+3$, which clips to $+2$, content term is $1.0 \times 1.2 = 1.2$, position term is $1.0 \times (-0.5) = -0.5$. Combined score: $0.7$.

The key insight is visible immediately: position 4, despite having the highest content similarity ($\mathbf{q}_1 \cdot \mathbf{k}_4 = 1.2$), is penalized by the relative position embedding because it is far away. Position 1 (same position) receives a slight boost. In a trained model, these biases would reflect linguistic preferences learned from data rather than arbitrary initialization values, but the mechanics are identical.

After applying softmax to the four scores $[1.0, 0.5, -0.1, 0.7]$, we get attention weights that favor the nearby positions more than pure content similarity would suggest. This is exactly the behavior we want: locality is naturally preferred, but strong content signals can still override the distance penalty when a truly relevant token appears far away.

Now that we understand the mathematical formulation and the worked example, let's translate it into code. We'll build up the implementation in stages: first the relative position embeddings, then the modified attention score computation, then the modified value aggregation, and finally the complete layer.

The foundation of Shaw-style attention is a table of learned embeddings indexed by relative position. Since relative positions range from $-k$ to $+k$ after clipping, we need $2k + 1$ embeddings. The key implementation detail is converting from relative positions (which can be negative) to array indices (which must be non-negative).

The index conversion is the subtle but essential detail. Relative positions range from $-k$ to $+k$, but array indices must be non-negative. Adding $k$ shifts the range to $[0, 2k]$:

• Offset $-k$ (maximum backward distance) maps to index 0
• Offset 0 (same position) maps to index $k$
• Offset $+k$ (maximum forward distance) maps to index $2k$

This simple arithmetic lets us use relative positions as array indices without any conditional logic. The same conversion applies uniformly across all offsets, making the lookup fast and predictable.

With the embedding table in place, we can implement the modified attention score computation. Recall the formula: $e_{ij} = (\mathbf{q}_i \cdot \mathbf{k}_j + \mathbf{q}_i \cdot \mathbf{a}^K_{j-i}) / \sqrt{d_k}$. For each query-key pair, we compute both the content term and the position term, then sum them before scaling.

This implementation iterates over all position pairs, making the logic transparent. For each pair $(i, j)$, we compute the content score ($\mathbf{q}_i \cdot \mathbf{k}_j$), look up the relative position embedding for offset $j - i$, compute the position score ($\mathbf{q}_i \cdot \mathbf{a}^K_{j-i}$), and sum them. The final division by $\sqrt{d_k}$ applies the same scaling as standard attention to prevent score magnitudes from growing with dimension.

The value aggregation follows the same pattern. For each output position $i$, we aggregate contributions from all positions $j$, but we add the relative position embedding to each value before weighting:

The value modification works similarly to the key modification. For each contributing position $j$, we add its relative position embedding $\mathbf{a}^V_{j-i}$ to its value vector before applying the attention weight. This means the output at position $i$ contains weighted content and weighted position information about where that content originated. The model can learn to tag information with its source distance, which helps downstream layers reason about the structure of the input.

With both components in place, we can assemble the complete layer. This class wraps the QKV projections, relative position embeddings, and the modified attention computation into a single forward pass:

The forward pass follows the standard self-attention pattern with two modifications. After projecting input embeddings to QKV, we use our modified relative_attention_scores function instead of the standard dot product, and we use relative_attention_output instead of the standard weighted sum. The softmax step remains unchanged, as the relative position terms are already incorporated into the scores before normalization.

Let's verify our implementation produces sensible results:

The embedding matrix shows how each relative offset is encoded as a different vector. Offset 0 (same position) has a distinct pattern from offset -3 (three positions back) or offset +3 (three positions forward). These patterns are learned during training to capture which distances are relevant for the task. After training on a language modeling task, you would typically see the nearby offsets ($\pm 1$, $\pm 2$) develop sharper, more structured patterns while the boundary offsets ($\pm k$) develop softer patterns reflecting their shared "far away" role.

Let's visualize how relative position embeddings influence attention. We'll examine the position term $\mathbf{q}_i \cdot \mathbf{a}^K_{j-i}$ separately from the content term. Isolating these two components lets us see how much of the final attention pattern comes from what tokens mean versus where they are.

The position biases subtly shift attention patterns. In this random example, the differences are modest because the embeddings are random. In trained models, relative position embeddings learn linguistically meaningful patterns: verbs attending strongly to positions 1-2 back (typical subject distance), determiners attending 1-2 forward (to their nouns), and so on. The combined plot differs from a sum of the two individual plots because the softmax nonlinearity mixes them: a moderate positive position bias for a nearby token can dramatically amplify its weight if the content score is also high, while a large positive bias for a distant token with low content score may still produce a negligible final weight.

The loop-based implementation above is clear but slow. In practice, we compute relative attention efficiently using matrix operations. The key insight is that position biases form a Toeplitz-like structure: all entries on each diagonal share the same relative distance.

The naive loop runs in $O(n^2 d_k)$ time, with a Python-level loop over all $n^2$ position pairs. The efficient version replaces the inner position term computation with a single matrix multiplication, reducing the Python overhead from $O(n^2)$ loop iterations to $O(1)$. The total asymptotic complexity remains $O(n^2 d_k)$, but the constant factor drops dramatically because NumPy's matrix operations run in compiled code.

The efficient version first computes all possible query-position dot products in a single $(n, 2k+1)$ matrix multiply, then indexes into this precomputed matrix. This reduces the inner loop computation from $O(d_k)$ per entry to $O(1)$, with the upfront cost of a single $(n, 2k+1)$ matrix multiplication. The Toeplitz structure of the position bias matrix is what makes this possible: once you know how every query interacts with every possible offset embedding, you only need to map indices to offsets, which is a pure indexing operation.

Real-world implementations in PyTorch or JAX go further. The indexing step can be replaced with a gather operation or einsum that exploits the diagonal structure, bringing the full bias computation down to a few vectorized calls with no Python loops at all. This is important for GPU utilization: Python loops are sequential, but GPU kernels can parallelize across the entire $(n, n)$ matrix simultaneously.

Let's directly compare relative and absolute position encodings on a simple task: detecting whether a token is attending to its immediate neighbors versus distant positions.

The profile shows how attention is distributed by relative distance. In this random initialization, patterns are noisy, but trained models develop clear preferences: immediate neighbors typically receive more attention, with the pattern decaying for distant positions. The key observation is that this profile is the same at every absolute position in the sequence. A token at position 50 and a token at position 500 both use the same distance-indexed biases. Absolute position encoding cannot achieve this: the bias for "position 50 attending to position 52" has no relationship to the bias for "position 500 attending to position 502," because the absolute embedding at position 52 is entirely independent from the absolute embedding at position 502.

Shaw et al.'s formulation has been adopted in several widely-used models, and understanding how those adoptions differ from the original paper helps calibrate when and how to use relative position encoding in your own work.

BERT and GPT variants originally used absolute position embeddings (either sinusoidal or learned), but researchers have since explored retrofitting relative position attention to these architectures. The main finding is that relative encoding consistently outperforms absolute encoding on tasks requiring cross-sentence reasoning or document-level understanding, where relevant information can appear at variable distances.

T5 uses a simplified form of relative position encoding that omits the value-side embeddings entirely and uses a smaller number of learned biases shared across all heads. This shared-bias design reduces the parameter count and works well for text-to-text generation tasks. T5's position biases are learned per-layer but shared across heads, a further simplification that incurs little performance cost.

Transformer-XL introduced segment-level recurrence to handle very long sequences, and paired it with a relative position encoding scheme based on sinusoidal functions rather than learned embeddings. Their formulation is more complex than Shaw et al.'s but avoids the need to re-initialize position embeddings when extending to new sequence lengths.

When implementing relative position encoding in practice, several configuration decisions matter:

• Clipping distance $k$: For sentence-level tasks, $k = 16$ or $k = 32$ is usually sufficient. For document-level tasks or tasks with long-range dependencies (coreference resolution, discourse structure), larger values like $k = 64$ or $k = 128$ are more appropriate.

• Value-side embeddings: Omitting $\mathbf{a}^V$ reduces parameters by half with modest performance loss. For memory-constrained settings or when training data is limited, this is a reasonable trade-off.

• Initialization scale: Relative position embeddings should be initialized with smaller values than QKV projections. A scale of $1/\sqrt{d_k}$ or the Xavier formula used in the code above prevents position biases from dominating attention scores at the start of training.

• Multi-head sharing: Some implementations share position embeddings across all heads (T5's approach), while others use per-head embeddings (Shaw et al.'s approach). Shared embeddings reduce parameters but limit each head's ability to specialize in different distance profiles.

The most important practical consideration is batching. In standard attention, you batch multiple sequences together and process them in parallel. With relative position encoding, all sequences in a batch share the same position embedding table, but the offset matrix for each sequence needs to be computed based on that sequence's length. This is straightforward for fixed-length sequences (pad to the same length) but requires careful handling for variable-length sequences with attention masking.

To illustrate what relative position embeddings might learn, let's simulate embeddings that encode common linguistic patterns:

This simulated pattern shows a Gaussian-like preference for nearby positions. Real models learn more complex patterns that capture linguistic structure: determiners attending to following nouns, verbs attending to preceding subjects, and so on. A trained model's position profiles are also typically asymmetric. This reflects the fact that English (and many other languages) has predominantly left-to-right dependency structure. Subjects appear before verbs, modifiers often precede their heads, and complementizers introduce subordinate clauses that follow the main verb. These directional biases show up clearly when you visualize position profiles from models trained on real text.

When implementing relative position encoding, several parameters control the behavior and efficiency of the mechanism. Getting these right matters more for relative encoding than for absolute encoding, because the position embeddings are learned and their training dynamics depend on initialization and capacity choices.

The key parameters are:

• max_distance (k): Maximum relative position to distinguish. Positions beyond this distance share the same "far away" embedding. Typical values range from 16 to 128. Smaller values reduce memory but lose fine-grained distance information for distant positions. Larger values preserve more detail but require more parameters and training data. The right value depends on the task: sentence-level tasks rarely need $k > 32$, while document-level tasks may benefit from $k = 128$.

• d_k: Dimension of query and key vectors, and key-side relative position embeddings. Must match between queries and keys for dot product compatibility. Common values are 64 for single-head attention or embed_dim / num_heads for multi-head attention. Larger $d_k$ gives the position embeddings more capacity to encode complex distance patterns.

• d_v: Dimension of value vectors and value-side relative position embeddings. Determines the output dimension of each attention head. Often set equal to d_k for simplicity, though they can differ.

• Embedding initialization scale: Relative position embeddings are typically initialized with small random values. Xavier/Glorot initialization scales by $\sqrt{2/(n\_\text{positions} + d)}$ to maintain stable gradient magnitudes during training. If initialization is too large, position biases dominate the attention scores from the start and the model struggles to learn content-based patterns. If too small, position information takes many epochs to develop.

The total number of relative position parameters is $(2k + 1) \times (d_k + d_v)$. With $k = 64$ and $d_k = d_v = 64$, this adds 16,512 parameters per attention head, which is modest compared to the QKV projection matrices (which have $3 \times d_{\text{model}} \times d_k$ parameters). In a 12-head transformer with $d_{\text{model}} = 768$, the QKV projections total about 1.77M parameters per layer, while relative position embeddings add only about 200K. The overhead is real but small.

Relative position encoding solves the generalization problem but introduces its own challenges. Understanding these limitations is important both for choosing between absolute and relative encoding in a new project and for anticipating where Shaw et al.'s approach will fall short.

The most significant limitation is computational complexity. Standard absolute position encoding adds to embeddings once at the input, with no additional cost during attention. Relative position encoding modifies every attention score computation. For a sequence of length $n$, we compute $n^2$ relative position biases per attention layer. With multiple layers and attention heads, this overhead accumulates. Shaw et al. note that their approach adds roughly 5-10% to training time, which is acceptable for most applications. However, for very long sequences (4096+ tokens), even a 10% overhead becomes significant because attention is already the bottleneck, and any additional computation in the $O(n^2)$ inner loop compounds quickly.

Memory usage also increases. We store $(2k+1)$ relative position embeddings per dimension, for both keys and values. With $d_k = d_v = 64$ and $k = 64$, this adds about 16K parameters per attention head. This is modest compared to the projection matrices, but the position bias matrix computed during the forward pass has shape $(n, n)$, which for $n = 4096$ requires 64 MB per layer just for the bias storage (in float32). For very long sequences, this memory footprint rivals the attention weight matrix itself.

The clipping distance $k$ introduces a hyperparameter choice with no universally correct answer. Too small, and the model loses fine-grained distance information for medium-range dependencies. Too large, and distant embeddings have insufficient training signal, particularly early in training when most gradients flow through frequent short-distance pairs. Shaw et al. used $k = 16$ in their original experiments on the WMT translation benchmark, but this value is too small for tasks involving long-range dependencies in English prose. In practice, $k$ is often set to match the typical context length during training, such as 64 or 128 for sentence-level tasks.

A subtler limitation is that Shaw et al.'s formulation encodes position at a single point in the transformer architecture: the attention score computation. This means position information can only influence the model through the routing decision (which tokens get attention) and the value aggregation (what information flows). It cannot influence the feedforward sublayers, layer norm operations, or any other component. More recent approaches like RoPE embed position information in a way that is more deeply integrated into the computation, allowing it to influence behavior throughout the network.

Finally, Shaw et al.'s relative position embeddings are fully learned, which means they depend on having sufficient training data to cover the full range of distance patterns the model needs to handle. For tasks with very limited data, the distant-position embeddings may remain undertrained throughout fine-tuning, creating a different kind of generalization problem than the one we set out to solve. In such cases, a fixed relative position scheme (like the sinusoidal approach used in Transformer-XL) may generalize better.

Despite these limitations, relative position encoding's impact on transformer research has been substantial. It demonstrated that position information can be injected at the attention level rather than the embedding level, opening the door to more sophisticated schemes like Rotary Position Embeddings and ALiBi that we'll explore in subsequent chapters. Every major language model architecture developed since 2019 has incorporated some form of relative position encoding, and the foundational ideas in this chapter appear in all of them.

Relative position encoding shifts the focus from "where am I?" to "how far apart are we?" By encoding the offset between positions rather than absolute indices, models learn patterns that generalize across all positions at the same distance.

Key takeaways from this chapter:

• Absolute positions limit generalization: A model trained on short sequences may never see position pairs that appear in longer sequences. Relative encoding ensures that distance patterns transfer regardless of absolute position. Nearly 75% of position pairs in a 2x-length test sequence are unseen during training on the shorter length.

• Shaw et al. formulation: Modify attention scores by adding $\mathbf{q}_i \cdot \mathbf{a}^K_{j-i}$ to the content term. This separates semantic compatibility from distance-based compatibility. The position term is a dot product between the query and the relative position embedding, allowing different query types to respond differently to the same distance.

• Clipping bounds memory: Relative positions beyond distance $k$ share the same embedding. This limits the number of parameters to $2k + 1$ regardless of sequence length, and also prevents distant embeddings from being undertrained due to infrequent occurrence.

• Two components to modify: Shaw et al. add relative embeddings to both keys (affecting which positions get attention) and values (affecting what content flows). Both contribute to the learned distance patterns, though many implementations omit the value-side for efficiency.

• Diagonal structure in attention: Position biases form a Toeplitz-like pattern where each anti-diagonal corresponds to a fixed relative distance. This structure enables efficient computation by precomputing all query-offset dot products in a single matrix multiply.

• Trade-off between complexity and generalization: Relative position encoding adds computational overhead (roughly 5-10% at training time) but provides better generalization to unseen position pairs and longer sequences. The overhead grows with sequence length, which has motivated subsequent methods designed for long-context settings.

In the next chapter, we'll explore Rotary Position Embeddings (RoPE), a method that encodes relative positions through rotation rather than additive biases. RoPE achieves similar generalization benefits with a more elegant mathematical formulation that integrates naturally with the attention computation, and it has become the dominant approach in large language models such as LLaMA and Mistral, along with GPT-NeoX.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about relative position encoding.