Rotary Position Embedding (RoPE) › The Complete RoPE Formula

Now that we understand the individual components, let's bring everything together into the complete RoPE transformation. We've established three key ideas:

1. Rotation encodes position: Each position $m$ corresponds to a rotation angle $m\theta$
2. Dot products extract relative position: When query and key are rotated by different amounts, their dot product depends only on the angle difference
3. Multiple frequencies create richness: Different dimension pairs rotate at different rates, capturing both local and global position information

The complete RoPE formula combines these insights into a single elegant operation. Given a query or key vector $\mathbf{x} \in \mathbb{R}^d$ at position $m$, we apply RoPE as follows:

$$\text{RoPE}(\mathbf{x}, m) = \begin{pmatrix} R(\theta_0 \cdot m) & & & \\ & R(\theta_1 \cdot m) & & \\ & & \ddots & \\ & & & R(\theta_{d/2-1} \cdot m) \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \\ \vdots \\ x_{d-1} \\ x_d \end{pmatrix}$$

where:

• $\text{RoPE}(\mathbf{x}, m)$: the rotated vector, a function of both the input vector and position
• $\mathbf{x} \in \mathbb{R}^d$: the input query or key vector with $d$ dimensions
• $m$: the sequence position (integer index)
• $R(\theta_i \cdot m)$: the 2$\times$2 rotation matrix for angle $\theta_i \cdot m$, applied to dimension pair $i$
• $\theta_i = 1 / 10000^{2i/d}$: the base frequency for dimension pair $i$ (decreases exponentially with $i$)
• The block-diagonal structure means each 2$\times$2 rotation block operates independently on its corresponding dimension pair
• Empty off-diagonal blocks are zeros, so dimensions in different pairs don't interact during rotation

The large block-diagonal matrix applies different rotations to each dimension pair simultaneously. This is efficient because each 2D rotation is independent of the others, allowing for parallel computation.

Expanding the rotation for a single dimension pair $(x_{2i+1}, x_{2i+2})$:

$$\begin{pmatrix} x'_{2i+1} \\ x'_{2i+2} \end{pmatrix} = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix} \begin{pmatrix} x_{2i+1} \\ x_{2i+2} \end{pmatrix}$$

where:

• $x_{2i+1}$, $x_{2i+2}$: the $(2i+1)$-th and $(2i+2)$-th components of the input vector (using 1-based indexing)
• $x'_{2i+1}$, $x'_{2i+2}$: the corresponding components after rotation
• $m\theta_i$: the rotation angle, which increases linearly with position $m$ at a rate determined by frequency $\theta_i$

Written out element-wise, the transformation is:

• $x'_{2i+1} = x_{2i+1} \cos(m\theta_i) - x_{2i+2} \sin(m\theta_i)$
• $x'_{2i+2} = x_{2i+1} \sin(m\theta_i) + x_{2i+2} \cos(m\theta_i)$