Rotary Position Embedding (RoPE) › Rotation Matrices in 2D

Michael BrenndoerferUpdated June 5, 20254 min read
Language AI HandbookAll chapters
Part 14 of 67Positional Encoding
Chapter 100Rotary Position Embedding (RoPE) › Rotation Matrices in 2D
Reading section 3 of 19 · Rotation Matrices in 2D · 4 minSection 3 of 19 · 4 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 100 of 513

Section 3 of 19 · 4 min read

Rotary Position Embedding (RoPE) › Rotation Matrices in 2DOne section · Sections in this chapter3 / 19 ↓
Sections in this chapter19
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To understand why RoPE works, you first need a firm grip on how rotation changes a vector and why the dot product between rotated vectors has a simple form. Let's build up the mechanics from scratch.

In two dimensions, rotating a vector (x,y)(x, y) by angle θ\theta counterclockwise uses the rotation matrix:

R(θ)=(cos⁡θ−sin⁡θsin⁡θcos⁡θ)R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}

where:

  • R(θ)R(\theta): the 2D rotation matrix that rotates vectors by angle θ\theta
  • θ\theta: the rotation angle in radians (counterclockwise is positive)
  • cos⁡θ\cos\theta, sin⁡θ\sin\theta: trigonometric functions evaluated at angle θ\theta

Applying this rotation matrix to a 2D vector transforms its coordinates:

(x′y′)=R(θ)(xy)=(xcos⁡θ−ysin⁡θxsin⁡θ+ycos⁡θ)\begin{pmatrix} x' \\ y' \end{pmatrix} = R(\theta) \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} x\cos\theta - y\sin\theta \\ x\sin\theta + y\cos\theta \end{pmatrix}

where:

  • (x,y)(x, y): the original vector coordinates
  • (x′,y′)(x', y'): the rotated vector coordinates
  • The first row computes the new xx-coordinate by combining the original coordinates with cos⁡θ\cos\theta and −sin⁡θ-\sin\theta
  • The second row computes the new yy-coordinate using sin⁡θ\sin\theta and cos⁡θ\cos\theta

The rotated vector (x′,y′)(x', y') has the same length as (x,y)(x, y) but points in a direction shifted by θ\theta. This is because rotation matrices are orthogonal, meaning they preserve vector lengths (norms) and angles between vectors. Orthogonality is the key property that makes everything else work: when you rotate both a query and a key, their magnitudes stay the same, and the only thing that changes in their dot product is the angular relationship between them.

In practice, two properties of orthogonal matrices matter most for RoPE. First, the transpose of a rotation matrix equals its inverse: R(θ)T=R(−θ)R(\theta)^T = R(-\theta). This tells you that "undoing" a rotation by angle θ\theta is the same as transposing the matrix, which will become important when we simplify the dot product expression. Second, multiplying two rotation matrices combines their angles: R(α)R(β)=R(α+β)R(\alpha) R(\beta) = R(\alpha + \beta). This composition rule is what allows absolute angles to cancel and leave only their difference.

In[4]:
Code
def rotate_2d(vector, theta):
    """Rotate a 2D vector by angle theta (in radians)."""
    cos_t, sin_t = np.cos(theta), np.sin(theta)
    rotation_matrix = np.array([[cos_t, -sin_t], [sin_t, cos_t]])
    return rotation_matrix @ vector

Let's visualize how rotation transforms a vector:

In[5]:
Code
# Original vector
original = np.array([1.0, 0.5])

# Rotate by several angles
angles = [0, np.pi / 6, np.pi / 3, np.pi / 2]
rotated_vectors = [rotate_2d(original, theta) for theta in angles]
Out[6]:
Visualization
Four arrows from origin showing a vector rotated by 0, 30, 60, and 90 degrees.
A 2D vector rotated by increasing angles. The vector maintains its length (distance from origin) while its direction changes. Each rotation of π/6 radians (30°) shifts the vector counterclockwise.

The dashed circle shows the path traced by the vector tip as it rotates. The length (magnitude) never changes. Rotation is an isometry, a transformation that preserves distances. This matters for attention: the magnitude of a query or key vector carries information about how strongly that token "wants" to attend or "wants" to be attended to. Rotation leaves that strength unchanged while shifting the directional relationship between the two vectors.