Rotary Position Embedding (RoPE) › Why Rotation?

Consider what we want from a position encoding. When a query at position $m$ attends to a key at position $n$, the attention score should somehow reflect their relative distance $m - n$. If token 3 attends to token 1, the model should "know" they're 2 positions apart, exactly as if token 8 attends to token 6.

Rotations have a beautiful property that accomplishes this. If you rotate vector $\mathbf{a}$ by angle $\theta_m$ and vector $\mathbf{b}$ by angle $\theta_n$, then compute their dot product, the result depends on the angle difference $\theta_m - \theta_n$. The absolute angles cancel out.

The key insight is that this invariance extends to the difference of rotations. If you rotate $\mathbf{a}$ by $\alpha$ and $\mathbf{b}$ by $\beta$, the dot product reflects the angular difference $\alpha - \beta$, not the individual angles. This is why rotation is a natural fit for relative position encoding: position becomes an angle, and the angle difference between query and key positions is exactly the relative separation we want the model to observe.

Now imagine we associate each position with an angle: position $m$ gets angle $m \cdot \theta$ for some base angle $\theta$. If we rotate the query vector at position $m$ by $m\theta$ and the key vector at position $n$ by $n\theta$, their dot product will involve the angle $(m - n)\theta$. That's exactly the relative position information we want.

Notice that no part of this requires changing the attention formula itself. Standard scaled dot-product attention simply computes $\mathbf{q}^T \mathbf{k} / \sqrt{d}$. RoPE just ensures that the vectors being dotted already carry the right geometric relationship. The attention mechanism does its job as usual, and position awareness comes for free.

This is the core insight of RoPE: encode position through rotation, and let the geometry of dot products naturally extract relative position.