Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

NTK-aware Scaling › Interpolation Factor Analysis

Michael BrenndoerferUpdated July 1, 20253 min read

Language AI HandbookAll chapters

Part 18 of 67Long Context

Chapter 128NTK-aware Scaling › Interpolation Factor Analysis

Reading section 9 of 14 · Interpolation Factor Analysis · 3 minSection 9 of 14 · 3 min

Section 9 of 14 · 3 min read

NTK-aware Scaling › Interpolation Factor AnalysisOne section · Sections in this chapter9 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

An alternative perspective on NTK-aware scaling is to analyze the interpolation factor for each frequency. Rather than thinking in terms of modified bases, we can ask: for each dimension pair, how far along the spectrum from "no change" to "fully interpolated" does NTK-aware scaling move us? This framing reveals that NTK-aware scaling is implicitly doing something very natural: it applies a smooth, dimension-dependent blend of the two extremes.

We can express NTK-aware scaling as a blend between "no interpolation" and "full interpolation":

In[27]:

Code

def compute_interpolation_factors(d, scale):
    """Compute effective interpolation factor for each dimension pair.

    Returns values between 0 (no interpolation, high freq preserved)
    and 1 (full interpolation).
    """
    dim_pairs = np.arange(d // 2)

    # NTK scaling formula: scale^(2i / (d-2))
    ntk_factors = scale ** (2 * dim_pairs / (d - 2))

    # Normalize to [0, 1] range: 1 means no scaling, scale means full scaling
    # interpolation = 0 when factor = 1, interpolation = 1 when factor = scale
    interpolation = (ntk_factors - 1) / (scale - 1)

    return interpolation

Out[28]:

Visualization

Line plot showing interpolation factor from 0 to 1 across dimension indices, with smooth S-curve transition. — Interpolation factor across dimension pairs under NTK-aware scaling. Low-frequency dimensions (right) receive full interpolation, while high-frequency dimensions (left) are preserved with minimal modification.

This interpolation perspective provides an intuitive understanding: NTK-aware scaling smoothly transitions from "preserve high frequencies" (interpolation factor near 0) to "fully scale low frequencies" (interpolation factor near 1).

The S-shaped curve in the interpolation factor plot reflects the exponential growth of the scaling factor. Initially, for the highest-frequency dimensions, the interpolation factor grows very slowly because the scaling factor is close to 1 and changing little. As you move to lower-frequency dimensions, the factor accelerates and approaches 1 more quickly. The exact shape depends on the scale factor $s$ : larger values of $s$ make the S-curve steeper, because the transition from near-zero to near-one must be completed in the same number of dimension pairs.

This perspective also helps you reason about what happens when you need an extension factor larger than what was originally intended. If a model was designed for $s = 4$ but you want to push to $s = 16$ , the interpolation factors at intermediate dimensions will be significantly larger, meaning more distortion in the middle frequencies. This is one reason why very large extension factors often require fine-tuning even when smaller extension factors can work zero-shot.