Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

YaRN › The YaRN Ramp Function

Michael BrenndoerferUpdated June 28, 20258 min read

Language AI HandbookAll chapters

Part 18 of 67Long Context

Chapter 129YaRN › The YaRN Ramp Function

Reading section 6 of 16 · The YaRN Ramp Function · 8 minSection 6 of 16 · 8 min

Section 6 of 16 · 8 min read

YaRN › The YaRN Ramp FunctionOne section · Sections in this chapter6 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Now we can design a function that decides how much interpolation each dimension pair receives. We want three behaviors, each motivated by the wavelength analysis above.

First, for pairs with small $r_i$ (short wavelengths, many rotations during training): apply no interpolation, leaving them at their original frequencies. These pairs are perfectly capable of distinguishing positions across the extended context without any modification.

Second, for pairs with large $r_i$ (long wavelengths, few rotations during training): apply full interpolation, scaling them by $1/s$ just like Position Interpolation would. These pairs must be slowed down so that the extended context positions fall within the rotation states the model has already learned.

Third, for pairs with intermediate $r_i$ : apply partial interpolation, blending smoothly between the two extremes. This smooth transition prevents discontinuities and allows the model to adapt gradually from the no-interpolation regime to the full-interpolation regime.

This is exactly what a ramp function achieves. Think of it as a dimmer switch rather than an on-off switch: instead of suddenly jumping from "no interpolation" to "full interpolation," it slides smoothly across a transition zone. YaRN defines $\gamma(r)$ as a piecewise function:

\gamma(r) = \begin{cases} 1 & \text{if } r < \alpha \\ \frac{1}{s} & \text{if } r > \beta \\ (1 - w) + w \cdot \frac{1}{s} & \text{otherwise} \end{cases}

where:

$r = \lambda_i / L$ : the wavelength-to-context ratio for dimension pair $i$ , comparing the rotation period to the training context length
$\lambda_i$ : the wavelength for dimension pair $i$
$L$ : the original training context length (e.g., 2048 or 4096 tokens)
$s$ : the extension scale factor (e.g., 4 for extending from 2048 to 8192 positions)
$\alpha$ : the lower threshold (typically 1.0), below which no interpolation is applied to the dimension pair
$\beta$ : the upper threshold (typically 32.0), above which full Position-Interpolation-style scaling is applied
$w = (r - \alpha) / (\beta - \alpha)$ : the interpolation weight in the ramp region, ranging linearly from 0 (at $r = \alpha$ ) to 1 (at $r = \beta$ )
$\gamma(r)$ : the output interpolation factor, ranging from 1 (no interpolation, frequency unchanged) to $1/s$ (full interpolation, frequency divided by the extension factor)

Let's unpack the middle case, which is the most interesting and distinguishes YaRN from simpler approaches. The weight $w$ measures where $r$ falls within the transition region $[\alpha, \beta]$ . At the left edge ( $r = \alpha$ ), we have $w = 0$ , so the expression becomes $(1 - 0) + 0 \cdot (1/s) = 1$ : the pair receives no interpolation. At the right edge ( $r = \beta$ ), we have $w = 1$ , giving $(1 - 1) + 1 \cdot (1/s) = 1/s$ : the pair receives full interpolation. For values between the edges, we get a linear blend of the two extremes. This creates a smooth ramp rather than an abrupt step, which helps the model adapt more gracefully across dimension pairs.

Why does this formula make sense? Consider what $\gamma(r) = 1$ means for the modified frequency: $\theta'_i = \theta_i \cdot 1 = \theta_i$ , unchanged. The pair rotates at its original speed, and positions in the extended context will be seen as extrapolated values. For high-frequency pairs with $r \ll 1$ , this is fine because they complete so many rotations that the model has already generalized to all rotation states. Consider what $\gamma(r) = 1/s$ means: $\theta'_i = \theta_i / s$ , which is exactly what Position Interpolation would apply if used globally. For low-frequency pairs with $r \gg 1$ , this scaling ensures that position $m$ in the extended context maps to the same rotation state as position $m/s$ in the original context.

Why

\alpha = 1

and

\beta = 32

?

The default thresholds have intuitive interpretations rooted in the wavelength analysis. When $\alpha = 1$ , dimension pairs with wavelength less than the context length (those completing at least one full rotation during training) are left unchanged. The model has visited all possible rotation states for these pairs and can handle any position without modification. When $\beta = 32$ , dimension pairs with wavelength more than 32 times the context length (completing less than 3% of a single rotation during training) receive full interpolation. These pairs are so slow-moving that they must be slowed down by the full extension factor to stay within familiar territory. The ramp between $\alpha = 1$ and $\beta = 32$ provides a smooth transition for the intermediate pairs that fall between these extremes. These values were empirically validated across multiple model architectures including LLaMA-2 and Mistral, and they strike a balance between preserving high-frequency information and ensuring stable extrapolation for low-frequency pairs.

To summarize, the ramp function creates three distinct regions:

Short wavelengths ( $r < \alpha$ ): These pairs already rotate many times within the original context. They don't need interpolation because they can naturally handle extended positions.
Long wavelengths ( $r > \beta$ ): These pairs rotate slowly and need full Position Interpolation treatment to avoid operating outside their training distribution.
Middle wavelengths ( $\alpha \leq r \leq \beta$ ): These pairs receive partial interpolation, smoothly blending between the two extremes.

In[11]:

Code

def yarn_gamma(wavelength, context_length, scale, alpha=1.0, beta=32.0):
    """Compute YaRN interpolation factor for a given wavelength."""
    r = wavelength / context_length

    if r < alpha:
        # Short wavelength: no interpolation
        return 1.0
    elif r > beta:
        # Long wavelength: full interpolation
        return 1.0 / scale
    else:
        # Ramp region: smooth transition
        t = (r - alpha) / (beta - alpha)
        # Linear interpolation between 1 and 1/scale
        return (1 - t) * 1.0 + t * (1.0 / scale)


def compute_yarn_frequencies(
    d_model, base=10000, context_length=2048, scale=4.0, alpha=1.0, beta=32.0
):
    """Compute YaRN-adjusted frequencies."""
    original_freqs = compute_rope_frequencies(d_model, base)
    wavelengths = compute_wavelengths(d_model, base)

    gamma_values = np.array(
        [yarn_gamma(w, context_length, scale, alpha, beta) for w in wavelengths]
    )

    # Apply gamma as a frequency multiplier
    # gamma < 1 means we slow down the rotation (interpolation)
    yarn_freqs = original_freqs * gamma_values

    return yarn_freqs, gamma_values

Let's visualize the ramp function and its effect on frequencies:

In[12]:

Code

# Compute YaRN frequencies
d_model = 64
context_length = 512
scale = 4.0

yarn_freqs, gamma_values = compute_yarn_frequencies(
    d_model, context_length=context_length, scale=scale
)
original_freqs = compute_rope_frequencies(d_model)
wavelengths = compute_wavelengths(d_model)

# Compute wavelength ratios
r_values = wavelengths / context_length

Out[13]:

Visualization

Line plot showing step-like gamma function with smooth ramp transition. — The YaRN gamma function plotted against wavelength ratio on a log scale. Pairs with wavelength ratios below alpha (1.0) receive no interpolation and gamma equals 1. Pairs with ratios above beta (32.0) receive full interpolation and gamma equals 0.25 for a 4x extension. The linear ramp in the log-wavelength region smoothly connects these two extremes, preventing sharp discontinuities in how adjacent dimension pairs are treated.

Log-scale plot comparing YaRN, PI, and NTK frequency spectra. — Effect of the ramp function on the frequency spectrum. YaRN (red) preserves high frequencies at the left side, matching the original, then transitions to interpolated frequencies on the right. This selective adjustment contrasts with Position Interpolation, which shifts all frequencies down uniformly, and NTK-aware scaling, which applies a smooth global bend to the entire spectrum.

The ramp function creates a piecewise linear transition on the log-wavelength scale. The left panel shows how $\gamma$ drops from 1 to $1/s$ as wavelength increases, with the smooth ramp connecting the two flat regions. The right panel shows the resulting frequency spectrum: YaRN preserves high frequencies at the left side of the dimension range (matching the original), then transitions smoothly to interpolated low frequencies on the right. This dimension-specific adjustment is the key innovation that sets YaRN apart from both PI and NTK-aware scaling.