YaRN › Wavelength Analysis

To understand why some dimension pairs need interpolation while others don't, we need to think about rotation in terms of wavelength rather than frequency. While frequency tells us how fast something rotates (radians per position), wavelength tells us how far we must travel for one complete cycle (positions per full rotation). Wavelength provides a more intuitive picture because we can directly compare it to the context length: if a wavelength is shorter than the context, the pair completes multiple full rotations during training; if longer, it completes only a fraction of one rotation.

Think of a dimension pair's rotation as a clock hand sweeping around a circle. A high-frequency pair is like a second hand: it completes many full rotations in the span of the training context. Every possible angle has been visited many times. The model knows how to use all parts of the rotation cycle to distinguish positions. A low-frequency pair is like an hour hand: it moves very slowly, covering only a small arc during training. The model has only seen a fraction of the rotation cycle, and it uses that small arc to encode position. If we extend the context beyond training, the second-hand pair has no problem: it just keeps spinning. But the hour-hand pair is now asked to cover territory it has never visited, and extrapolation fails.

The wavelength $\lambda_i$ of dimension pair $i$ is the inverse of the frequency, scaled by $2\pi$:

$$\lambda_i = \frac{2\pi}{\theta_i} = 2\pi \cdot 10000^{2i/d}$$

where:

• $\lambda_i$: the wavelength (in positions) for dimension pair $i$, representing how many sequence positions correspond to one complete rotation cycle
• $\theta_i = 1/10000^{2i/d}$: the base frequency for dimension pair $i$, measured in radians per position
• $d$: the total embedding dimension (must be even, since RoPE operates on pairs)
• $i$: the dimension pair index, ranging from $0$ to $d/2 - 1$
• $2\pi$: the number of radians in a complete rotation (one full cycle)
• $10000$: the RoPE base constant, which controls the overall frequency range across dimensions

The second equality follows from substituting the definition of $\theta_i$ and simplifying: $2\pi / (1/10000^{2i/d}) = 2\pi \cdot 10000^{2i/d}$.

Why does this formula make sense? Notice that as $i$ increases (moving to higher dimension pair indices), the exponent $2i/d$ grows from 0 toward 2, making $10000^{2i/d}$ grow from 1 toward $10000^2 = 10^8$. So wavelengths grow exponentially with dimension pair index: the first pairs have very short wavelengths (high frequencies), and the last pairs have very long wavelengths (low frequencies). This exponential spread is what gives RoPE its ability to encode position at multiple scales simultaneously.

The critical question becomes: how does each wavelength compare to the training context length $L$? If $\lambda_i < L$, the pair completes at least one full rotation during training, meaning the model has seen all possible rotation states. If $\lambda_i > L$, the pair completes less than one rotation, meaning the model has only seen a fraction of the rotation cycle. The ratio $r_i = \lambda_i / L$ is central to YaRN's design: it tells us, for each dimension pair, how much of the rotation cycle the model has already explored.

The "vs Original L" column shows the wavelength ratio $r_i = \lambda_i / L$. This ratio is the key to understanding which dimension pairs need interpolation:

• When $r_i < 1$, the wavelength is shorter than the context, meaning the pair completes multiple full rotations during training. These pairs have learned the full rotation cycle and can handle extended positions without modification.
• When $r_i > 1$, the wavelength exceeds the context, meaning the pair does not complete even one rotation during training. These pairs operate in a limited portion of the rotation cycle and need interpolation to avoid extrapolation.

Let's trace through the table to build concrete intuition. Dimension pair 0 has a wavelength of about 6 positions and $r_0 \approx 0.003$. It completes roughly 340 full rotations within the training context, so it has thoroughly learned how to use rotation for position encoding across its entire cycle. When we extend to 8,192 positions, pair 0 still completes over 1,300 rotations. No problem here: it is operating well within familiar territory.

At the other extreme, dimension pair 31 has a wavelength of around 60,000 positions and $r_{31} \approx 29$. During training, it completes only about 3% of a single rotation. The model has learned to encode position using just this small arc of the rotation cycle. If we extend the context by 4x without interpolation, we would ask pair 31 to cover 12% of its rotation cycle, using rotation states it has never encountered during training. This is extrapolation, and it systematically degrades model quality.

The pattern is clear: pairs with small $r_i$ do not need interpolation; pairs with large $r_i$ do. The question is where to draw the line, and whether the transition should be abrupt or gradual. An abrupt threshold would create a discontinuity in how different dimension pairs are treated, which could introduce artifacts in the attention computation. A smooth transition, on the other hand, allows adjacent dimension pairs to behave similarly, which YaRN achieves with its ramp function.