YaRN › The Problem with Existing Methods
Language AI HandbookAll chapters YaRN › The Problem with Existing MethodsOne section · Sections in this chapter2 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into YaRN, let's understand precisely why Position Interpolation and NTK-aware scaling leave room for improvement. Both methods successfully enable RoPE-based models to process longer sequences, but they introduce subtle distortions that accumulate as the extension factor increases. Understanding these distortions motivates every design choice in YaRN.
Rotary Position Embedding (RoPE) encodes position by rotating token embeddings in the complex plane. Each pair of embedding dimensions is assigned a rotation frequency , ranging from high frequencies (rapid rotation, distinguishing nearby tokens) to low frequencies (slow rotation, distinguishing distant tokens). The key property of RoPE is that the dot product between a query at position and a key at position depends only on their relative offset . This is what gives transformer models their relative position sensitivity.
Position Interpolation works by scaling all position indices by a factor : a position becomes . This keeps all positions within the trained range, but it compresses the entire frequency spectrum uniformly. High-frequency dimensions that previously distinguished positions 1 apart now need to distinguish positions apart after compression. The model loses fine-grained position discrimination in all frequency bands simultaneously. Think of it as zooming out on a photograph: every detail shrinks by the same factor, and fine details become indistinguishable. For small extension factors (1.5x or 2x), the blurring is tolerable. For larger factors, the high-frequency bands lose so much precision that nearby tokens begin to look indistinguishable.
NTK-aware scaling addresses this by adjusting the base frequency rather than scaling positions directly. This preserves high-frequency components while stretching low-frequency ones. The approach works well for moderate extensions (2x to 4x), but at larger factors, even NTK-aware scaling begins to degrade. The root issue is that NTK-aware scaling still applies the same mathematical transformation uniformly across all dimension pairs, just with a different parameterization. Some dimension pairs benefit from modification; others would be better left alone. Neither PI nor NTK provides the dimension-specific control that optimal extension requires.
There is also a second limitation that both methods share: neither accounts for the entropy change that interpolation introduces into attention distributions. Modifying position encodings changes how similar queries look to keys across different positions. When positions appear compressed, all keys look more similar to every query, and the softmax over attention scores becomes flatter. This entropy increase is not a feature. It is a side effect that degrades the model's ability to focus on specific relevant tokens. YaRN is the first method to address this problem explicitly and correct for it with a principled formula.
import numpy as np
def compute_rope_frequencies(d_model, base=10000):
"""Compute standard RoPE rotation frequencies."""
num_pairs = d_model // 2
i = np.arange(num_pairs)
frequencies = 1.0 / (base ** (2 * i / d_model))
return frequencies
def compute_pi_frequencies(d_model, base=10000, scale=4.0):
"""Position Interpolation: scale all positions by 1/scale."""
frequencies = compute_rope_frequencies(d_model, base)
return frequencies / scale
def compute_ntk_frequencies(d_model, base=10000, scale=4.0):
"""NTK-aware scaling: adjust base instead of positions.
The new base is computed as: base * scale^(d / (d - 2))
This increases the base, which decreases all frequencies,
but preserves the high-frequency components better than PI.
"""
new_base = base * scale ** (d_model / (d_model - 2))
return compute_rope_frequencies(d_model, new_base)Let's visualize how these methods transform the frequency spectrum:
d_model = 64
scale = 4.0
original_freqs = compute_rope_frequencies(d_model)
pi_freqs = compute_pi_frequencies(d_model, scale=scale)
ntk_freqs = compute_ntk_frequencies(d_model, scale=scale)
The visualization reveals a fundamental trade-off. Position Interpolation maintains the relative spacing between frequencies (the lines are parallel on a log scale) but shifts everything down uniformly. NTK-aware scaling bends the curve, keeping high frequencies close to original while aggressively stretching low frequencies. Neither approach is optimal across all dimension pairs, because they apply a global transformation where a dimension-specific one is needed.