Relative Position Encoding › Worked Example: Step-by-Step Score Computation

Let's walk through a concrete, minimal example to make the math tangible. Consider a four-token sequence with one-dimensional embeddings, maximum clipping distance $k = 2$, and very simple relative position embeddings. The goal is to show exactly where the relative bias comes from and how it modifies the attention scores.

Suppose we have queries $\mathbf{q}_1 = [1.0]$, $\mathbf{q}_2 = [0.5]$, $\mathbf{q}_3 = [0.8]$, $\mathbf{q}_4 = [0.3]$ and keys $\mathbf{k}_1 = [0.9]$, $\mathbf{k}_2 = [0.7]$, $\mathbf{k}_3 = [0.4]$, $\mathbf{k}_4 = [1.2]$. The key-side relative position embeddings (after clipping to $k = 2$) are:

• $\mathbf{a}^K_{-2} = [-0.3]$ (two positions back)
• $\mathbf{a}^K_{-1} = [+0.4]$ (one position back)
• $\mathbf{a}^K_{0} = [+0.1]$ (same position)
• $\mathbf{a}^K_{+1} = [-0.2]$ (one position forward)
• $\mathbf{a}^K_{+2} = [-0.5]$ (two positions forward)

Now let's compute the score from query position 1 to all key positions. With $d_k = 1$ so $\sqrt{d_k} = 1$:

For position pair $(i=1, j=1)$: offset is $0$, content term is $1.0 \times 0.9 = 0.9$, position term is $1.0 \times 0.1 = 0.1$. Combined score: $1.0$.

For position pair $(i=1, j=2)$: offset is $+1$, content term is $1.0 \times 0.7 = 0.7$, position term is $1.0 \times (-0.2) = -0.2$. Combined score: $0.5$.

For position pair $(i=1, j=3)$: offset is $+2$, content term is $1.0 \times 0.4 = 0.4$, position term is $1.0 \times (-0.5) = -0.5$. Combined score: $-0.1$.

For position pair $(i=1, j=4)$: offset is $+3$, which clips to $+2$, content term is $1.0 \times 1.2 = 1.2$, position term is $1.0 \times (-0.5) = -0.5$. Combined score: $0.7$.

The key insight is visible immediately: position 4, despite having the highest content similarity ($\mathbf{q}_1 \cdot \mathbf{k}_4 = 1.2$), is penalized by the relative position embedding because it is far away. Position 1 (same position) receives a slight boost. In a trained model, these biases would reflect linguistic preferences learned from data rather than arbitrary initialization values, but the mechanics are identical.

After applying softmax to the four scores $[1.0, 0.5, -0.1, 0.7]$, we get attention weights that favor the nearby positions more than pure content similarity would suggest. This is exactly the behavior we want: locality is naturally preferred, but strong content signals can still override the distance penalty when a truly relevant token appears far away.