Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

ALiBi › ALiBi vs. RoPE: A Comparison

Michael BrenndoerferUpdated May 30, 20254 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 101ALiBi › ALiBi vs. RoPE: A Comparison

Reading section 10 of 16 · ALiBi vs. RoPE: A Comparison · 4 minSection 10 of 16 · 4 min

Section 10 of 16 · 4 min read

ALiBi › ALiBi vs. RoPE: A ComparisonOne section · Sections in this chapter10 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Both ALiBi and RoPE are widely used in modern language models, and both address relative position. But they take fundamentally different approaches that lead to different trade-offs in practice.

RoPE encodes position by rotating query and key vectors. When computing dot products between rotated vectors, the rotation angles combine such that the result depends on relative position rather than absolute position. This is mathematically elegant: the dot product $\mathbf{q}_i \cdot \mathbf{k}_j$ after rotation naturally depends on $i - j$ rather than on $i$ and $j$ separately. RoPE modifies the embedding space itself: the vectors that participate in dot products carry rotational information baked into their components.

ALiBi's approach is different in character: it does not modify the vectors at all. The dot products $\mathbf{q}_i \cdot \mathbf{k}_j$ remain the raw content-based scores. ALiBi then overlays a separate, explicit position signal on top of those scores. The two components are additive and fully transparent. You can look at a score and immediately decompose it into a content part (the dot product) and a position part (the bias).

ALiBi vs. RoPE comparison across key dimensions.
Aspect	ALiBi	RoPE
Position encoding location	Attention score bias	Query and key embeddings
Mechanism	Subtracts linear penalty from attention scores	Rotates Q and K vectors by position-dependent angles
Parameters	Fixed slopes, no learned position parameters	No additional parameters
Computation	Simple matrix addition to score matrix	Complex number arithmetic or 2D rotation matrices
Extrapolation	Strong out-of-the-box at long sequences	Requires additional techniques (NTK scaling, interpolation)
Interpretability	Fully transparent: slope directly controls window width	Less direct: rotation angle interplay is harder to interpret
Impact on embeddings	None: token embeddings are pure semantic vectors	Embeds absolute rotation into token representations

The choice between ALiBi and RoPE often comes down to empirical performance on the target task. RoPE has shown excellent results on many benchmarks and is used by prominent models including LLaMA and Mistral, as well as Falcon. ALiBi has shown strong extrapolation and is used by BLOOM and MPT. Newer research (including models like LLaMA 3 and Mixtral) tends to favor RoPE with context extension techniques, suggesting that RoPE's expressiveness may ultimately win out as practitioners develop better tools for extending its context window. However, ALiBi remains a practical choice when implementation simplicity and extrapolation matter, especially when minimizing implementation risk.

Let's compare the implementation complexity concretely, which is where ALiBi's advantage is clearest:

In[22]:

Code

def rope_attention(Q, K, V, seq_len, d_k, causal=True):
    """
    Simplified RoPE attention for comparison.
    This is a sketch showing the additional complexity.
    """
    # RoPE requires computing rotation matrices or using complex numbers
    # For each position, apply rotation to Q and K before computing attention

    # Step 1: Compute rotation angles for each position and dimension
    positions = np.arange(seq_len)
    dim_indices = np.arange(d_k // 2)

    # Frequency for each dimension pair (simplified)
    freqs = 1.0 / (10000 ** (2 * dim_indices / d_k))

    # Angle matrix: (seq_len, d_k/2)
    angles = positions[:, None] * freqs[None, :]

    # Step 2: Apply rotation to Q and K (complex number approach)
    # This involves reshaping Q and K, applying cos/sin transformations...
    # (full implementation omitted for brevity)

    # The key point: RoPE modifies the embeddings themselves
    # before attention computation
    pass


def alibi_attention_simple(Q, K, V, slopes, causal=True):
    """
    ALiBi attention for comparison.
    """
    # Step 1: Standard attention scores
    scores = np.matmul(Q, K.transpose(0, 2, 1)) / np.sqrt(Q.shape[-1])

    # Step 2: Add distance-based bias (one line!)
    seq_len = Q.shape[1]
    distances = np.abs(
        np.arange(seq_len)[:, None] - np.arange(seq_len)[None, :]
    )
    scores = scores + (-slopes[:, None, None] * distances)

    # That's it. Apply softmax and compute output.
    return scores

The ALiBi bias is a single line of code added to standard attention. The entire position encoding logic lives in one array operation. RoPE requires restructuring how queries and keys are computed, introducing trigonometric functions and careful handling of dimension pairs. Both work, but ALiBi's simplicity is a practical advantage: fewer lines of code means fewer places for bugs to hide, and the position bias is always directly visible and inspectable.