YaRN › The YaRN Solution

YaRN combines two complementary mechanisms to enable high-quality context extension. Neither mechanism is sufficient on its own: the frequency adjustment alone does not fix the entropy problem, and the temperature correction alone does not fix the frequency distortion. But together, they address both sources of degradation in a way that is both principled and computationally lightweight.

The first mechanism is ramp-based frequency interpolation. Rather than treating all dimension pairs equally (like PI) or smoothly transitioning across all pairs using a single global parameter (like NTK), YaRN uses a ramp function that applies no interpolation to high-frequency pairs, full interpolation to low-frequency pairs, and a smooth linear transition in between. The threshold between "needs interpolation" and "does not need interpolation" is based on whether the pair's rotation wavelength fits within the original training context. This makes the interpolation decision interpretable and motivated by the actual frequency behavior of each dimension.

The second mechanism is attention temperature scaling. YaRN introduces a scaling factor $\sqrt{t}$ applied to attention scores before softmax, where $t \geq 1$ is computed from the extension factor $s$ via an empirically derived formula. This scaling increases the spread of attention scores, counteracting the compression effect of interpolation. When $s = 1$ (no extension), $t = 1$ and no scaling is applied, preserving the original behavior exactly.

The combined approach modifies both the rotation frequencies and the attention computation. For frequencies, YaRN applies a dimension-specific adjustment. The core idea is to compute a modified frequency $\theta'_i$ for each dimension pair $i$:

$$\theta'_i = \theta_i \cdot \gamma(r_i)$$

where:

• $\theta'_i$: the adjusted frequency for dimension pair $i$ after YaRN modification
• $\theta_i$: the original RoPE frequency for dimension pair $i$, computed as $1/10000^{2i/d}$
• $\gamma(r_i)$: the interpolation factor, a value between $1/s$ and $1$ that depends on the wavelength ratio for this dimension
• $r_i = \lambda_i / L$: the wavelength-to-context ratio for dimension pair $i$, comparing this pair's rotation period to the training context length
• $\lambda_i$: the wavelength for dimension pair $i$, measuring how many sequence positions correspond to one full rotation
• $L$: the original training context length (e.g., 2048 or 4096 tokens)

Why does this formula make sense? Notice that when $\gamma(r_i) = 1$, the adjusted frequency equals the original frequency: no change. When $\gamma(r_i) = 1/s$, the adjusted frequency is scaled down by exactly the extension factor, which is equivalent to Position Interpolation applied to just this dimension. The ramp function interpolates between these extremes based on how much this particular frequency needs correction.

For attention scores, YaRN introduces a temperature correction:

$$\text{score}'(m, n) = \sqrt{t} \cdot \frac{\mathbf{q}_m^\top \mathbf{k}_n}{\sqrt{d_k}}$$

where:

• $\text{score}'(m, n)$: the temperature-adjusted attention score between positions $m$ and $n$
• $\sqrt{t}$: the temperature scaling factor, where $t > 1$ increases score magnitude and sharpens the attention distribution
• $t$: the temperature parameter, computed from the extension factor $s$ via $t = 0.1 \ln(s) + 1$
• $\mathbf{q}_m$, $\mathbf{k}_n$: the YaRN-rotated query and key vectors at positions $m$ and $n$
• $d_k$: the dimension of key vectors, used for the standard $\sqrt{d_k}$ scaling

The scaling factor $\sqrt{t}$ appears before the softmax, not after. Multiplying pre-softmax scores by a constant $c$ sharpens the distribution (for $c > 1$) in the same way that reducing temperature does in generative sampling. YaRN uses this mechanism to counteract the entropy increase from frequency interpolation.

Let's develop each component in detail.