Sinusoidal Position Encoding › The Relative Position Property

One of the most elegant properties of sinusoidal encoding is that relative positions can be expressed as linear transformations. For any fixed offset $k$, there exists a matrix $M_k$ such that:

where:

• $PE_{pos}$: the position encoding vector at position $pos$ (a $d$-dimensional row vector)
• $PE_{pos+k}$: the position encoding vector at position $pos + k$
• $M_k$: a $d \times d$ transformation matrix that depends only on the offset $k$, not on the absolute position

This is a remarkable property. It means that the relationship between any two encodings that are $k$ steps apart is always the same, regardless of where they sit in the sequence. The transformation from position 5 to position 10 is the same linear map as the transformation from position 500 to position 505. In principle, a transformer that learns this linear map can attend to any token "k positions ahead" without needing to know the absolute positions, simply by applying a fixed linear operation to the encoding.

To understand why this works, we need to derive the relationship step by step using trigonometric identities.

Step 1: Recall the angle addition formulas. For any angles $a$ and $b$, trigonometry gives us:

where $a$ and $b$ are any angles (in radians). These identities let us express the sine or cosine of a sum in terms of the sines and cosines of the individual angles. They provide the algebra behind the relative position construction.

Step 2: Define the angular frequency. For dimension pair $i$, we define the angular frequency as:

where:

• $\omega_i$: the angular frequency for dimension pair $i$ (determines how fast this dimension oscillates)
• $i$: the dimension pair index (0, 1, 2, ..., $d/2 - 1$)
• $d$: the total embedding dimension
• $10000^{2i/d}$: the denominator that grows geometrically with $i$

This means the encoding at position $pos$ in dimension pair $i$ uses the argument $\omega_i \cdot pos$.

Step 3: Apply the addition formulas. To find the encoding at position $pos + k$, we substitute $a = \omega_i \cdot pos$ and $b = \omega_i \cdot k$ into the angle addition formulas:

where:

• $\omega_i = 1/10000^{2i/d}$: the angular frequency for dimension pair $i$
• $pos$: the current position in the sequence
• $k$: the position offset we want to shift by
• $\sin(\omega_i \cdot pos)$ and $\cos(\omega_i \cdot pos)$: the original encoding values at position $pos$ (these are $PE_{(pos, 2i)}$ and $PE_{(pos, 2i+1)}$)
• $\sin(\omega_i k)$ and $\cos(\omega_i k)$: constants that depend only on the offset $k$, not on the absolute position

The last two terms are fixed constants once you fix $k$ and $i$: the values $\sin(\omega_i k)$ and $\cos(\omega_i k)$ do not change as you vary $pos$. This is what makes the transformation linear in the original encoding values.

Step 4: Recognize the matrix structure. The key insight is that the right-hand sides of both equations are linear combinations of $\sin(\omega_i \cdot pos)$ and $\cos(\omega_i \cdot pos)$. This is exactly what matrix multiplication does. We can write:

This is a rotation in 2D. For each dimension pair, the encoding at $pos + k$ is the encoding at $pos$ rotated by angle $\omega_i k$. The rotation matrix for offset $k$ in dimension pair $i$ is:

where:

• $R_k^{(i)}$: the 2x2 rotation matrix for dimension pair $i$ with offset $k$
• $\omega_i = 1/10000^{2i/d}$: the angular frequency for dimension pair $i$
• $k$: the position offset (how many positions to shift)
• $\omega_i k$: the rotation angle, which depends on both the offset and the dimension's frequency

Step 5: Construct the full transformation matrix. The full transformation matrix $M_k$ is block-diagonal, with each 2x2 block being the rotation matrix for that dimension pair:

where:

• $M_k$: the full $d \times d$ transformation matrix for offset $k$
• $R_k^{(i)}$: the 2x2 rotation matrix for dimension pair $i$ (defined above)
• $0$: 2x2 zero matrices (indicating no interaction between dimension pairs)
• The matrix has $d/2$ blocks along the diagonal

This block-diagonal structure means relative position shifts act independently on each dimension pair, rotating the (sine, cosine) pair by an amount proportional to the offset. Each dimension pair encodes position at its own frequency, so shifting by $k$ positions rotates it by its characteristic angle $\omega_i k$. The faster the dimension's frequency (smaller $i$), the larger the rotation for a given offset $k$.

This rotation interpretation is more than a mathematical curiosity. It means that the inner product between two position encodings, $PE_{pos_1} \cdot PE_{pos_2}$, depends only on their relative distance $|pos_1 - pos_2|$, not on their absolute values. The attention mechanism computes inner products between queries and keys, both of which have position information added. If the model learns to use that position information, it can compute relative-position-sensitive attention scores. This is the theoretical basis for the claim that sinusoidal encoding supports relative position reasoning.

Let's verify this property numerically:

The errors are at machine precision, confirming that the relative position property holds exactly. This mathematical structure is what allows transformers to potentially learn relative position relationships through their linear attention projections.

Let's visualize this rotation property for a single dimension pair. We'll show how applying the rotation matrix to an encoding at position $pos$ produces the encoding at position $pos + k$:

The visualization makes the rotation property tangible. Each colored arrow shows the transformation from position $pos$ (circle) to position $pos + 5$ (square). Notice that all arrows rotate by the same angle, confirming that the transformation depends only on the offset $k$, not on the starting position. This is the geometric essence of how sinusoidal encodings enable learning of relative positions.