Relative Position Encoding › From Absolute to Relative: The Key Insight

Recall the standard self-attention formula. Given queries $\mathbf{Q}$, keys $\mathbf{K}$, and values $\mathbf{V}$, we compute:

$$\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V}$$

where:

• $\mathbf{Q} \in \mathbb{R}^{n \times d_k}$: query matrix containing all query vectors
• $\mathbf{K} \in \mathbb{R}^{n \times d_k}$: key matrix containing all key vectors
• $\mathbf{V} \in \mathbb{R}^{n \times d_v}$: value matrix containing all value vectors
• $d_k$: dimension of queries and keys
• $\sqrt{d_k}$: scaling factor to prevent dot products from growing too large
• $n$: sequence length

With absolute position encoding, each token's embedding includes position information before the QKV projections. Position is baked into the input: $\mathbf{x}_i = \mathbf{e}_i + \mathbf{p}_i$, where $\mathbf{e}_i$ is the token embedding and $\mathbf{p}_i$ is the position encoding. The attention scores then implicitly depend on absolute positions through the projected queries and keys. The model sees position information, but only indirectly: it must disentangle "what is this token?" from "where is this token?" inside the projection matrices themselves.

Relative position encoding takes a different approach: inject position information directly into the attention computation. Instead of adding position to the input, we modify how attention scores are calculated to explicitly account for the offset $j - i$ between positions $i$ and $j$. This is a cleaner separation of concerns. Content similarity handles semantic matching; a separate position-dependent term handles distance-based preferences. The model does not need to learn to disentangle them because they are never mixed in the first place.

The core idea is to add a position-dependent term to the attention score. To give the model a sense of why tokens at certain distances should receive more or less attention, we write:

$$\text{score}_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j + \text{relative\_bias}(i, j)$$

where:

• $\mathbf{q}_i$: the query vector at position $i$
• $\mathbf{k}_j$: the key vector at position $j$
• $\text{relative\_bias}(i, j)$: a term that depends only on the offset $j - i$

This formulation separates content matching ($\mathbf{q}_i \cdot \mathbf{k}_j$) from position matching (the relative bias). The model can learn that certain offsets are generally more or less important, regardless of absolute position. Notice that the relative bias only depends on the difference $j - i$, not on $i$ and $j$ individually. This is the key mathematical property that enables generalization: two position pairs with the same offset get the same positional treatment, no matter where they appear in the sequence.