Rotary Position Embedding (RoPE) › Worked Example: Step-by-Step RoPE for a 4-Dimensional Vector

Before we look at batched implementations, it helps to trace through a complete example by hand. This makes the abstract rotation formula concrete and shows exactly what changes when you apply RoPE to a real vector.

Suppose we have a 4-dimensional query vector at position $m = 2$:

$$\mathbf{q} = (1.0,\ 0.5,\ -0.8,\ 0.3)$$

With $d = 4$, we have two dimension pairs and two frequencies:

$$\theta_0 = \frac{1}{10000^{0/4}} = 1.0, \qquad \theta_1 = \frac{1}{10000^{2/4}} = \frac{1}{100} = 0.01$$

At position $m = 2$, the rotation angles for each pair are:

• Pair 0: $\phi_0 = 2 \times \theta_0 = 2 \times 1.0 = 2.0$ radians
• Pair 1: $\phi_1 = 2 \times \theta_1 = 2 \times 0.01 = 0.02$ radians

Now apply the 2D rotation formula to each pair separately. For pair 0, with $(x_1, x_2) = (1.0, 0.5)$ and $\phi_0 = 2.0$:

$$\begin{aligned} x'_1 &= x_1 \cos(\phi_0) - x_2 \sin(\phi_0) \\ &= 1.0 \times \cos(2.0) - 0.5 \times \sin(2.0) \\ &= 1.0 \times (-0.416) - 0.5 \times (0.909) \\ &= -0.416 - 0.455 = -0.871 \\[6pt] x'_2 &= x_1 \sin(\phi_0) + x_2 \cos(\phi_0) \\ &= 1.0 \times 0.909 + 0.5 \times (-0.416) \\ &= 0.909 - 0.208 = 0.701 \end{aligned}$$

For pair 1, with $(x_3, x_4) = (-0.8, 0.3)$ and $\phi_1 = 0.02$ (small angle, so $\cos(0.02) \approx 1.0$ and $\sin(0.02) \approx 0.02$):

$$\begin{aligned} x'_3 &= -0.8 \times 1.0 - 0.3 \times 0.02 = -0.8 - 0.006 = -0.806 \\ x'_4 &= -0.8 \times 0.02 + 0.3 \times 1.0 = -0.016 + 0.3 = 0.284 \end{aligned}$$

The rotated query is $\mathbf{q}' = (-0.871,\ 0.701,\ -0.806,\ 0.284)$. Notice that the first pair changed dramatically (2 radians is a large rotation), while the second pair barely moved (0.02 radians is about 1 degree). This illustrates the multi-scale nature of RoPE: high-frequency pairs carry a strong position signal that varies quickly from token to token, while low-frequency pairs change slowly and provide a coarser sense of position.

To confirm the key property, you would perform the same calculation for a key vector at position $n = 5$, compute the dot product $\mathbf{q}'^T \mathbf{k}'$, and verify that it equals the dot product you would get at any other pair of positions with relative distance $5 - 2 = 3$. The numbers work out exactly because the composition of rotations always produces the relative angle regardless of where in the sequence you started.