Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Rotary Position Embedding (RoPE) › From Rotation to Relative Position

Michael BrenndoerferUpdated June 5, 20253 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 100Rotary Position Embedding (RoPE) › From Rotation to Relative Position

Reading section 4 of 19 · From Rotation to Relative Position · 3 minSection 4 of 19 · 3 min

Section 4 of 19 · 3 min read

Rotary Position Embedding (RoPE) › From Rotation to Relative PositionOne section · Sections in this chapter4 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Now let's see how rotation makes dot products depend on relative position. Take two 2D vectors $\mathbf{q}$ (query) and $\mathbf{k}$ (key). Rotate $\mathbf{q}$ by angle $m\theta$ (for position $m$ ) and $\mathbf{k}$ by angle $n\theta$ (for position $n$ ).

The dot product of the rotated vectors is:

R(m\theta)\mathbf{q} \cdot R(n\theta)\mathbf{k}

where:

$\mathbf{q}$ : the query vector at position $m$
$\mathbf{k}$ : the key vector at position $n$
$R(m\theta)$ : rotation matrix that rotates by angle $m\theta$ (position times base angle)
$R(n\theta)$ : rotation matrix that rotates by angle $n\theta$

Using properties of rotation matrices, we can simplify this expression step by step:

\mathbf{q}^T R(m\theta)^T R(n\theta) \mathbf{k} = \mathbf{q}^T R(-m\theta) R(n\theta) \mathbf{k} = \mathbf{q}^T R((n-m)\theta) \mathbf{k}

The derivation proceeds as follows:

Rewrite the dot product as matrix multiplication: $R(m\theta)\mathbf{q} \cdot R(n\theta)\mathbf{k} = (R(m\theta)\mathbf{q})^T (R(n\theta)\mathbf{k}) = \mathbf{q}^T R(m\theta)^T R(n\theta) \mathbf{k}$
Apply the transpose-inverse property: The transpose of a rotation matrix equals its inverse, so $R(\theta)^T = R(-\theta)$ . This gives us $R(m\theta)^T = R(-m\theta)$ .
Apply the composition property: Multiplying two rotation matrices adds their angles, so $R(\alpha) R(\beta) = R(\alpha + \beta)$ . Therefore $R(-m\theta) R(n\theta) = R((n-m)\theta)$ .

The final result $\mathbf{q}^T R((n-m)\theta) \mathbf{k}$ depends only on the difference $(n-m)$ , not on the absolute values of $m$ and $n$ separately.

Key Insight: Relative Position Emerges

When we rotate query by $m\theta$ and key by $n\theta$ , their dot product depends only on $(n - m)\theta$ , the relative position. The absolute positions $m$ and $n$ vanish, replaced by their difference.

This is remarkable. We encode absolute position through rotation angle, but the attention mechanism, which uses dot products, automatically extracts relative position. No architectural changes needed. No explicit bias terms. Just geometry.

The derivation also reveals something that's easy to overlook: the original vectors $\mathbf{q}$ and $\mathbf{k}$ are still present in the final expression $\mathbf{q}^T R((n-m)\theta) \mathbf{k}$ . Their content still matters, it's just that the rotation wraps that content in a position-dependent frame. The model can still learn that "verb tokens should attend to their subject nouns"; it just does so in a coordinate system where position is already factored out. This is a cleaner separation of concerns than additive encodings, where position and content signals are entangled from the start.

Let's verify this numerically:

In[7]:

Code

def verify_relative_position():
    """Verify that rotated dot products depend only on relative position."""
    # Random query and key vectors
    q = np.random.randn(2)
    k = np.random.randn(2)

    theta = 0.5  # Base rotation angle

    # Different absolute positions, same relative distance
    results = []
    for m, n in [(1, 3), (5, 7), (10, 12), (100, 102)]:
        q_rotated = rotate_2d(q, m * theta)
        k_rotated = rotate_2d(k, n * theta)
        dot_product = np.dot(q_rotated, k_rotated)
        results.append((m, n, n - m, dot_product))

    return results


position_results = verify_relative_position()

Out[8]:

Console

Verifying relative position property:
Position m   Position n   n - m      Dot Product    
--------------------------------------------------
1            3            2          -0.651890
5            7            2          -0.651890
10           12           2          -0.651890
100          102          2          -0.651890

All four pairs have the same relative distance (2 positions apart), and their dot products are identical despite wildly different absolute positions. This confirms the relative position property holds numerically. Notice that positions 1 and 3, 5 and 7, 10 and 12, and 100 and 102 all produce the same score, exactly as the geometric derivation predicted.