Relative Position Encoding › Shaw et al. Relative Position Representations

The most influential formulation of relative positions in self-attention comes from Shaw et al. (2018). Their approach asks a fundamental question: if we want attention to be distance-aware, where exactly should we inject position information? Rather than modifying the input embeddings, Shaw et al. discovered that injecting relative positions directly into the attention computation gives the model finer control over how distance affects both which tokens get attention and what information flows.

To understand their approach, we need to build up three connected ideas: what we want the model to learn about distance, how to represent those distances as learnable parameters, and how to integrate those representations into the attention mechanism itself.

Consider what happens when position $i$ attends to position $j$ in standard self-attention. The model computes a score based on content similarity, then uses that score to weight how much of $j$'s information flows to $i$. But we want distance to influence both of these operations independently.

Think of it this way: a verb attending to possible subjects has two separate questions to answer. First, which position is the most likely subject based on both content and distance? Second, once the subject is found, how much of its information should flow into the verb's representation? Both questions depend on distance, but in different ways. The model might want to assign high attention to a subject that is two positions back, and to weight that subject's contribution with a "subject-attended-from-two-back" color that differs from "subject-attended-from-ten-back."

Shaw et al. address both needs by introducing two sets of learnable embeddings: one that modifies attention scores (controlling who gets attention), and one that modifies value aggregation (controlling what information flows). This separation gives the model independent control over "who to attend to" versus "what to gather."

1. Which positions get attention: A verb might prefer to attend to positions 1-3 positions back (where subjects typically appear), regardless of content similarity.

2. What information flows: When aggregating information, we might want "nearby context" to carry different weight than "distant context," even when attention weights are equal.

For each possible offset $r = j - i$ between positions, Shaw et al. define learnable vectors that the model adjusts during training just like any other parameter:

• $\mathbf{a}^K_r \in \mathbb{R}^{d_k}$: a relative position embedding that influences attention scores (key-side)
• $\mathbf{a}^V_r \in \mathbb{R}^{d_v}$: a relative position embedding that modifies aggregated values (value-side)

where:

• $r = j - i$: the relative offset (positive means looking forward, negative means looking backward)
• $d_k$: dimension of query and key vectors
• $d_v$: dimension of value vectors

These embeddings are learned during training, just like the QKV projection matrices. The critical insight is that the number of distinct offsets is bounded by the sequence length (from $-(n-1)$ to $+(n-1)$), not by the number of position pairs (which is $n^2$). This means the model learns a single pattern for "three positions back" that applies universally, rather than learning separate patterns for every pair of absolute positions. The parameter count scales linearly with the clipping distance $k$, not quadratically with sequence length.

The key-side embeddings ($\mathbf{a}^K$) and value-side embeddings ($\mathbf{a}^V$) are separate parameter matrices, not tied to each other. This gives the model full flexibility: the same distance can produce different effects on the routing decision (score computation) versus the aggregation decision (value weighting).

Let's trace through how relative positions enter the attention computation. In standard self-attention, the score measuring how much position $i$ should attend to position $j$ captures only content similarity:

$$e_{ij} = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}}$$

where:

• $e_{ij}$: the raw attention score from query position $i$ to key position $j$
• $\mathbf{q}_i \in \mathbb{R}^{d_k}$: the query vector at position $i$
• $\mathbf{k}_j \in \mathbb{R}^{d_k}$: the key vector at position $j$
• $d_k$: the dimension of query and key vectors

This score depends only on the content at each position. Two identical words at different positions produce identical scores, regardless of their distance. The model cannot distinguish "I saw a cat (right here, adjacent)" from "I saw a cat (twenty words ago)."

Shaw et al. modify this by adding the relative position embedding to the key before computing the dot product:

$$e_{ij} = \frac{\mathbf{q}_i \cdot (\mathbf{k}_j + \mathbf{a}^K_{j-i})}{\sqrt{d_k}}$$

Why add to the key rather than the query? The key represents "what this position offers," so adding position information to it means the model can learn that positions at certain distances offer something different, even if their content is identical. The query remains focused on "what I'm looking for" without distance context; the key communicates "here is what I have, adjusted for how far away I am from the query."

Expanding the dot product reveals the two-component structure:

$$e_{ij} = \frac{\mathbf{q}_i \cdot \mathbf{k}_j + \mathbf{q}_i \cdot \mathbf{a}^K_{j-i}}{\sqrt{d_k}}$$

where:

• $\mathbf{q}_i \cdot \mathbf{k}_j$: the content term, measuring semantic compatibility between positions
• $\mathbf{q}_i \cdot \mathbf{a}^K_{j-i}$: the position term, encoding how relative distance affects the score
• $\mathbf{a}^K_{j-i} \in \mathbb{R}^{d_k}$: the learned relative position embedding for offset $(j-i)$

The position term is a dot product between the query and the relative position embedding. This means different queries can respond differently to the same distance. A query learned to find subjects might have a high dot product with $\mathbf{a}^K_{-2}$ (two positions back), while a query learned to find objects might prefer $\mathbf{a}^K_{+1}$ (one position forward). The model learns these preferences end-to-end from training data, so the resulting position patterns reflect actual linguistic structure rather than any hand-designed prior.

Shaw et al. apply the same principle to value aggregation. After computing attention weights $\alpha_{ij} = \text{softmax}(e_{ij})$, standard attention aggregates values as a weighted sum. We want to add relative position information here too, so that the output at each position carries the weighted content of attended tokens and a record of where that content came from.

The modified value aggregation is:

$$\mathbf{o}_i = \sum_{j=1}^{n} \alpha_{ij} (\mathbf{v}_j + \mathbf{a}^V_{j-i})$$

where:

• $\mathbf{o}_i \in \mathbb{R}^{d_v}$: output vector at position $i$, now enriched with position-aware context
• $\alpha_{ij}$: attention weight from position $i$ to position $j$ (sums to 1 across $j$)
• $\mathbf{v}_j \in \mathbb{R}^{d_v}$: value vector at position $j$, carrying the content to aggregate
• $\mathbf{a}^V_{j-i} \in \mathbb{R}^{d_v}$: relative position embedding for values at offset $(j-i)$
• $n$: sequence length

This modification means the output at each position incorporates the weighted content from other positions and position-dependent information about where that content came from. Even if two positions have identical values and identical attention weights, their contributions differ based on their relative distances. Think of it as: "I got this information from a token that was three positions back, and that proximity matters for how I should interpret it."

The two sets of embeddings ($\mathbf{a}^K$ and $\mathbf{a}^V$) work together but serve distinct roles. The key embeddings affect which positions receive attention (the "routing" decision). The value embeddings affect what information flows once routing is decided (the "content" decision). This separation allows the model to learn, for example, that adjacent positions should receive high attention (key embedding effect) while also learning that adjacent context should contribute a specific "locality" signal to the output (value embedding effect).

In practice, many implementations omit the value-side embeddings ($\mathbf{a}^V$) for efficiency reasons. The key-side modification tends to deliver most of the benefit, and removing the value modification cuts the number of relative position parameters in half. The T5 model, for instance, uses only key-side biases. Shaw et al.'s original paper includes both, but their ablation experiments showed that omitting the value-side embeddings costs only a small amount of task performance.