Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Rotary Position Embedding (RoPE) › Complex Number Perspective

Michael BrenndoerferUpdated June 5, 20254 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 100Rotary Position Embedding (RoPE) › Complex Number Perspective

Reading section 7 of 19 · Complex Number Perspective · 4 minSection 7 of 19 · 4 min

Section 7 of 19 · 4 min read

Rotary Position Embedding (RoPE) › Complex Number PerspectiveOne section · Sections in this chapter7 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The matrix formulation we've developed is mathematically complete, but complex numbers provide another way to express RoPE. This notation reveals the connection between rotations and exponentials and leads to more efficient implementations.

The key insight is that a 2D rotation is equivalent to multiplication by a complex exponential. Every 2D vector $(x, y)$ can be viewed as a complex number $z = x + jy$ , where $j$ is the imaginary unit. In this representation, rotating the vector by angle $\theta$ is simply multiplying by $e^{j\theta}$ .

Think of it this way: the complex plane is just the 2D plane with an algebraic structure layered on top. The real axis is the $x$ -axis and the imaginary axis is the $y$ -axis. A complex number $z = x + jy$ is just the point $(x, y)$ , and multiplying two complex numbers combines their magnitudes and adds their angles. This multiplicative structure is exactly what we need: composing rotations corresponds to multiplying complex exponentials, and the angle-addition law $e^{j\alpha} \cdot e^{j\beta} = e^{j(\alpha+\beta)}$ is the algebraic counterpart of the rotation composition rule $R(\alpha)R(\beta) = R(\alpha+\beta)$ .

For the dimension pair $(x_{2i+1}, x_{2i+2})$ , interpret them as the real and imaginary parts of a complex number:

z_i = x_{2i+1} + j \cdot x_{2i+2}

where:

$z_i$ : a complex number representing dimension pair $i$
$x_{2i+1}$ : the real part (first element of the pair)
$x_{2i+2}$ : the imaginary part (second element of the pair)
$j$ : the imaginary unit (satisfying $j^2 = -1$ )

Rotation by angle $\theta$ in the complex plane is achieved by multiplication:

z'_i = z_i \cdot e^{j\theta} = z_i \cdot (\cos\theta + j\sin\theta)

where:

$z'_i$ : the rotated complex number
$e^{j\theta}$ : the complex exponential, a point on the unit circle at angle $\theta$
$\cos\theta + j\sin\theta$ : the expanded form via Euler's formula

Euler's Formula

Euler's formula states that $e^{j\theta} = \cos\theta + j\sin\theta$ . Geometrically, $e^{j\theta}$ represents a point on the unit circle at angle $\theta$ from the positive real axis. Multiplying any complex number by $e^{j\theta}$ rotates it by angle $\theta$ counterclockwise in the complex plane, preserving its magnitude.

For RoPE at position $m$ , we apply this rotation with the position-dependent angle:

z'_i = z_i \cdot e^{j \cdot m \cdot \theta_i}

where:

$m$ : the sequence position
$\theta_i$ : the base frequency for dimension pair $i$
$m \cdot \theta_i$ : the total rotation angle (increases linearly with position)

This formulation is mathematically equivalent to the rotation matrix approach. The complex perspective leads to more concise code and can be more efficient on hardware with optimized complex number operations. PyTorch, for instance, provides native complex tensor support with torch.view_as_complex, which allows the rotation to be expressed as a single element-wise multiplication rather than explicit matrix-vector products. This is why most production RoPE implementations use the complex formulation even when the underlying mathematics is easier to explain via rotation matrices.

In[13]:

Code

def rope_complex(x, position, frequencies):
    """Apply RoPE using complex number formulation.

    Args:
        x: Input vector of shape (d,) where d is even
        position: Position index (integer)
        frequencies: Pre-computed frequencies of shape (d/2,)

    Returns:
        Rotated vector of shape (d,)
    """
    # Reshape to pairs and view as complex numbers
    d = len(x)
    x_pairs = x.reshape(-1, 2)
    x_complex = x_pairs[:, 0] + 1j * x_pairs[:, 1]

    # Rotation angles for this position
    angles = position * frequencies

    # Apply rotation via complex multiplication
    rotated_complex = x_complex * np.exp(1j * angles)

    # Convert back to real pairs
    rotated_pairs = np.stack(
        [rotated_complex.real, rotated_complex.imag], axis=1
    )
    return rotated_pairs.flatten()

Let's verify that the complex formulation gives the same result as explicit rotation matrices:

In[14]:

Code

def rope_matrix(x, position, frequencies):
    """Apply RoPE using explicit rotation matrices."""
    d = len(x)
    result = np.zeros_like(x)

    for i in range(d // 2):
        theta = position * frequencies[i]
        cos_t, sin_t = np.cos(theta), np.sin(theta)

        # Apply 2D rotation to dimension pair (2i, 2i+1)
        x1, x2 = x[2 * i], x[2 * i + 1]
        result[2 * i] = x1 * cos_t - x2 * sin_t
        result[2 * i + 1] = x1 * sin_t + x2 * cos_t

    return result


# Compare both implementations
test_vector = np.random.randn(8)
test_freqs = compute_rope_frequencies(8)
position = 5

result_complex = rope_complex(test_vector, position, test_freqs)
result_matrix = rope_matrix(test_vector, position, test_freqs)

Out[15]:

Console

Comparing RoPE implementations:
Original vector:     [-0.2342 -0.2341  1.5792  0.7674 -0.4695  0.5426 -0.4634 -0.4657]
Complex formulation: [-0.2909  0.1581  1.018   1.4306 -0.496   0.5184 -0.4611 -0.468 ]
Matrix formulation:  [-0.2909  0.1581  1.018   1.4306 -0.496   0.5184 -0.4611 -0.468 ]

Max difference: 5.55e-17

The two implementations produce identical results (up to floating-point precision). The complex formulation is often preferred in practice because it is more concise and can use optimized complex number operations.