Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LLaMA Components › Rotary Position Embedding (RoPE)

Michael BrenndoerferUpdated August 2, 202512 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 208LLaMA Components › Rotary Position Embedding (RoPE)

Reading section 4 of 12 · Rotary Position Embedding (RoPE) · 12 minSection 4 of 12 · 12 min

Section 4 of 12 · 12 min read

LLaMA Components › Rotary Position Embedding (RoPE)One section · Sections in this chapter4 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Position information in LLaMA comes from Rotary Position Embedding, applied directly to query and key vectors before attention computation. Unlike additive position encodings that modify token embeddings once at the input, RoPE encodes position through rotation at every attention layer.

To understand why LLaMA chose RoPE over simpler alternatives, we need to appreciate a basic tension in position encoding: we want the model to understand both where tokens are in a sequence and how far apart they are from each other. Absolute position matters for tasks like "what's the first word?", but relative position matters for language understanding: the relationship between a verb and its subject depends on their distance, not their absolute positions.

Previous approaches added position information to token embeddings at the input layer. This works but has drawbacks: the position signal can degrade through deep networks, and the model must learn to extract relative position from absolute encodings. RoPE solves this elegantly by encoding position in a way that naturally produces relative position information when computing attention scores.

The Rotation PrincipleLink Copied

The key insight behind RoPE comes from a beautiful property of rotations. Imagine two arrows (vectors) on a piece of paper. If you rotate both arrows by different amounts and then measure the angle between them, you'll find something remarkable: the angle between them depends only on how much more you rotated one than the other. The absolute rotation amounts don't matter, only their difference.

This is exactly what we want for position encoding. If we rotate the query vector by an amount proportional to its position $m$ , and rotate the key vector by an amount proportional to its position $n$ , then when we compute their dot product (which measures similarity), the result depends only on the difference $m - n$ . The absolute positions cancel out, leaving pure relative position information.

Out[11]:

Visualization

2D plane showing arrows (vectors) rotated by different amounts based on their position, showing how RoPE encodes position through rotation. — Geometric intuition for RoPE: rotating vectors by position-dependent angles. The query at position 4 is rotated 4 times as much as a query at position 1. When computing attention scores, only the angular difference matters, naturally encoding relative position.

For a $d$ -dimensional embedding, RoPE treats it as $d/2$ independent 2D pairs. Think of each pair as a separate 2D plane where we can apply rotation. Each pair rotates at a different frequency, creating a multi-scale representation: some dimensions capture fine-grained local position differences, while others track global position across the entire sequence.

The rotation angle for dimension pair $i$ at position $m$ determines how much to rotate that pair. Different dimension pairs rotate at exponentially different rates, following a formula inspired by the original sinusoidal position encodings:

\theta_i(m) = m \cdot \theta_i = m \cdot \frac{1}{10000^{2i/d}}

where:

$\theta_i(m)$ : the rotation angle (in radians) for dimension pair $i$ at sequence position $m$
$m$ : sequence position (0, 1, 2, ...), with $m = 0$ for the first token
$i$ : dimension pair index (0, 1, ..., $d/2 - 1$ ), grouping dimensions into pairs
$d$ : total embedding dimension (must be even for RoPE)
$\theta_i = 1/10000^{2i/d}$ : base frequency for dimension pair $i$ , which decreases exponentially as $i$ increases
$10000$ : base constant chosen empirically (same as in sinusoidal position encodings)

The term $2i/d$ in the exponent creates the multi-scale effect: when $i = 0$ , $\theta_0 = 1$ (fast rotation, one radian per position); when $i$ approaches $d/2$ , $\theta_i$ approaches $1/10000$ (very slow rotation). This exponential spacing ensures that the model can distinguish positions at multiple scales simultaneously.

Now we can write the complete RoPE transformation. Given a vector $\mathbf{x}$ at position $m$ , we rotate each pair of dimensions $(x_{2i+1}, x_{2i+2})$ by angle $\theta_i \cdot m$ . The rotation uses the standard 2D rotation formula you might remember from linear algebra: to rotate a point $(x, y)$ by angle $\theta$ , you compute $(x\cos\theta - y\sin\theta, x\sin\theta + y\cos\theta)$ . Applying this to all dimension pairs gives:

\text{RoPE}(\mathbf{x}, m) = \begin{pmatrix} x_1 \cos(\theta_0 m) - x_2 \sin(\theta_0 m) \\ x_1 \sin(\theta_0 m) + x_2 \cos(\theta_0 m) \\ x_3 \cos(\theta_1 m) - x_4 \sin(\theta_1 m) \\ x_3 \sin(\theta_1 m) + x_4 \cos(\theta_1 m) \\ \vdots \end{pmatrix}

where:

$\mathbf{x} = (x_1, x_2, x_3, x_4, \ldots, x_d)^T$ : the input query or key vector
$(x_1, x_2)$ : the first dimension pair, rotated by angle $\theta_0 m$
$(x_3, x_4)$ : the second dimension pair, rotated by angle $\theta_1 m$
$\cos(\theta_i m), \sin(\theta_i m)$ : rotation components for pair $i$ at position $m$

Each pair is rotated independently using the standard 2D rotation formulas: $x' = x\cos\theta - y\sin\theta$ and $y' = x\sin\theta + y\cos\theta$ . This preserves the vector's magnitude while encoding position through angle. The preservation of magnitude is important: we're adding position information without distorting the semantic content of the embedding.

In practice, we don't need to construct explicit rotation matrices. Instead, we can compute the rotations efficiently using element-wise operations. The key observation is that for each dimension pair, we just need to:

Compute the angle $\theta_i \cdot m$ for that pair at position $m$
Calculate $\cos$ and $\sin$ of the angle
Apply the rotation formula to the pair

Let's implement this step by step:

In[12]:

Code

def compute_rope_frequencies(dim, base=10000):
    """Compute RoPE frequency for each dimension pair."""
    freqs = 1.0 / (base ** (np.arange(0, dim, 2) / dim))
    return freqs


def apply_rope(x, freqs, positions):
    """
    Apply rotary position embedding to input tensor.

    Args:
        x: Input tensor of shape (seq_len, dim)
        freqs: Frequency tensor of shape (dim/2,)
        positions: Position indices of shape (seq_len,)

    Returns:
        Rotated tensor of shape (seq_len, dim)
    """
    seq_len, dim = x.shape

    # Compute angles for each position and frequency
    angles = positions[:, None] * freqs[None, :]  # (seq_len, dim/2)

    # Compute sin and cos
    cos = np.cos(angles)
    sin = np.sin(angles)

    # Split input into pairs
    x_even = x[:, 0::2]  # Even dimensions
    x_odd = x[:, 1::2]  # Odd dimensions

    # Apply rotation to each pair
    x_rotated_even = x_even * cos - x_odd * sin
    x_rotated_odd = x_even * sin + x_odd * cos

    # Interleave back
    output = np.zeros_like(x)
    output[:, 0::2] = x_rotated_even
    output[:, 1::2] = x_rotated_odd

    return output

Let's visualize how RoPE creates position-dependent rotations across different frequencies:

Out[13]:

Visualization

Heatmap showing rotation angles for different dimension pairs across sequence positions, with faster oscillations for lower dimension indices. — RoPE rotation patterns across positions and dimension pairs. Early dimension pairs (low index) rotate quickly, capturing local position differences. Later dimension pairs rotate slowly, encoding global position information.

The heatmap reveals RoPE's multi-scale nature. Dimension pairs near the top (low index) oscillate rapidly: they complete many cycles across the sequence and can distinguish nearby positions precisely. Dimension pairs near the bottom (high index) change slowly: they differentiate positions that are far apart. Together, they create a rich position encoding that works at all scales.

RoPE in AttentionLink Copied

RoPE is applied to query and key vectors before computing attention scores. Critically, it is not applied to values. To understand why, consider what happens when we compute the attention score between a query at position $m$ and a key at position $n$ .

Without RoPE, the dot product between query $\mathbf{q}$ and key $\mathbf{k}$ gives:

\text{score} = \mathbf{q}^T \mathbf{k}

With RoPE, both vectors are rotated before the dot product. For a single 2D pair, if we rotate $\mathbf{q}$ by angle $\theta m$ and $\mathbf{k}$ by angle $\theta n$ , the dot product becomes:

\text{score} = \mathbf{q}^T R(\theta m)^T R(\theta n) \mathbf{k} = \mathbf{q}^T R(\theta(n - m)) \mathbf{k}

where:

$\mathbf{q}, \mathbf{k} \in \mathbb{R}^2$ : query and key vectors for a single dimension pair
$R(\theta)$ : the 2D rotation matrix $\begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$
$R(\theta)^T = R(-\theta)$ : the transpose of a rotation matrix is its inverse (rotation by the negative angle)
$\theta m, \theta n$ : rotation angles at positions $m$ and $n$ respectively

The rotation identity shows why only relative position remains: $R(\theta m)^T R(\theta n) = R(-\theta m) R(\theta n) = R(\theta(n - m))$ . The absolute positions $m$ and $n$ cancel out, leaving only their difference $n - m$ . This is how RoPE encodes relative position: the attention score between two positions depends only on how far apart they are, not where they appear in the sequence.

Out[14]:

Visualization

Heatmap showing attention score modulation based on relative position, with diagonal bands showing position-dependent attention patterns. — RoPE creates attention patterns that depend on relative position. This heatmap shows the positional contribution to attention scores: each row represents a query position, and each column a key position. The pattern depends only on the distance between positions, creating the characteristic diagonal structure.

Values don't participate in this relative position mechanism; they simply carry the information to be aggregated based on the position-aware attention weights.

In[15]:

Code

class RoPEAttention:
    """
    Multi-head self-attention with Rotary Position Embedding.

    Applies RoPE to queries and keys to encode relative positions.
    """

    def __init__(self, d_model, n_heads, base=10000):
        assert d_model % n_heads == 0

        self.d_model = d_model
        self.n_heads = n_heads
        self.d_head = d_model // n_heads

        # Projections
        scale = np.sqrt(2.0 / (d_model + d_model))
        self.W_q = np.random.randn(d_model, d_model) * scale
        self.W_k = np.random.randn(d_model, d_model) * scale
        self.W_v = np.random.randn(d_model, d_model) * scale
        self.W_o = np.random.randn(d_model, d_model) * scale

        # RoPE frequencies (per head dimension)
        self.freqs = compute_rope_frequencies(self.d_head, base)

    def __call__(self, x, mask=None):
        """
        Args:
            x: Input of shape (seq_len, d_model)
            mask: Optional causal mask of shape (seq_len, seq_len)

        Returns:
            Output of shape (seq_len, d_model)
        """
        seq_len = x.shape[0]
        positions = np.arange(seq_len)

        # Project to Q, K, V
        Q = x @ self.W_q  # (seq_len, d_model)
        K = x @ self.W_k
        V = x @ self.W_v

        # Reshape to (seq_len, n_heads, d_head)
        Q = Q.reshape(seq_len, self.n_heads, self.d_head)
        K = K.reshape(seq_len, self.n_heads, self.d_head)
        V = V.reshape(seq_len, self.n_heads, self.d_head)

        # Apply RoPE to each head's Q and K
        for h in range(self.n_heads):
            Q[:, h, :] = apply_rope(
                Q[:, h, :].reshape(seq_len, self.d_head), self.freqs, positions
            )
            K[:, h, :] = apply_rope(
                K[:, h, :].reshape(seq_len, self.d_head), self.freqs, positions
            )

        # Compute attention scores
        # (seq_len, n_heads, d_head) @ (d_head, n_heads, seq_len) per head
        scores = np.zeros((seq_len, self.n_heads, seq_len))
        for h in range(self.n_heads):
            scores[:, h, :] = Q[:, h, :] @ K[:, h, :].T / np.sqrt(self.d_head)

        # Apply causal mask if provided
        if mask is not None:
            scores = np.where(mask[:, None, :], scores, -1e9)

        # Softmax over keys
        scores_max = scores.max(axis=-1, keepdims=True)
        exp_scores = np.exp(scores - scores_max)
        attention = exp_scores / exp_scores.sum(axis=-1, keepdims=True)

        # Apply attention to values
        output = np.zeros((seq_len, self.n_heads, self.d_head))
        for h in range(self.n_heads):
            output[:, h, :] = attention[:, h, :] @ V[:, h, :]

        # Reshape and project output
        output = output.reshape(seq_len, self.d_model)
        return output @ self.W_o