NTK-aware Scaling › The Problem with Uniform Scaling

To understand why uniform scaling falls short, we need to revisit how RoPE frequencies work and appreciate what each part of the frequency spectrum contributes to the model's ability to encode position. Uniform scaling causes measurable degradation. Understanding that degradation helps us design better solutions.

Recall that RoPE assigns each dimension pair $i$ a base frequency that determines how fast the rotation angle changes with position:

$$\theta_i = \frac{1}{b^{2i/d}}$$

where:

• $\theta_i$: the base rotation frequency (in radians per position) for dimension pair $i$
• $b$: the base constant, typically 10000, which controls the overall frequency range
• $i$: the dimension pair index, ranging from 0 to $d/2 - 1$
• $d$: the total embedding dimension (must be even since we work in pairs)
• $2i/d$: the exponent that creates a geometric progression of frequencies across dimensions

The first dimension pair ($i = 0$) has frequency $\theta_0 = 1/10000^0 = 1$, meaning it rotates by 1 radian per position and completes a full rotation every $2\pi \approx 6.28$ positions. The last pair ($i = d/2 - 1$) has a much smaller frequency, completing a rotation only after thousands of positions. This spread creates a multi-scale representation:

• High-frequency pairs ($i$ near 0): Distinguish nearby tokens with precision. Tokens at positions 5 and 6 look very different.
• Low-frequency pairs ($i$ near $d/2$): Capture coarse position information. Tokens far apart have noticeably different rotations.

Notice that this multi-scale design is deliberate and important. RoPE's ability to encode position without explicit positional embeddings depends on the entire spectrum working together. Each frequency band plays a role in shaping the inner products between queries and keys in the attention mechanism. When you attend from position $m$ to position $n$, the resulting attention score is influenced by the rotation difference $\Delta = n - m$ across every dimension pair simultaneously. High-frequency pairs create a rapidly oscillating component that is sensitive to small values of $\Delta$, while low-frequency pairs create a slowly varying component that only changes significantly when $\Delta$ is large.

Position Interpolation addresses context extension by scaling all frequencies uniformly:

$$\theta_i' = \frac{\theta_i}{s}$$

where:

• $\theta_i'$: the scaled frequency for dimension pair $i$ after Position Interpolation
• $\theta_i$: the original frequency for dimension pair $i$
• $s$: the context extension factor, computed as $s = L_{\text{target}} / L_{\text{train}}$ (target length divided by training length)

If we want to extend from 4,096 to 32,768 tokens, $s = 32768 / 4096 = 8$. Every frequency gets divided by 8. The high-frequency pair that previously rotated 1 radian per position now rotates only $1/8 = 0.125$ radians. Adjacent tokens, which were once clearly distinguishable, become nearly indistinguishable in these dimensions.

The key insight is that this is a lossy compression. The model was trained to use the high-frequency dimensions to encode fine-grained neighborhood relationships. Tokens at positions 100 and 101 should look clearly different in those dimensions, because the attention mechanism needs to distinguish "the word immediately to my right" from "the word two positions to my right." After Position Interpolation with $s = 8$, those two positions differ by only $0.125$ radians instead of 1 radian, a factor of 8 reduction in separability. The model has no way to recover this information; it simply cannot see a distinction that was previously obvious.

In practice, this translates to measurable performance degradation. Tasks that depend heavily on local attention, such as identifying named entity boundaries, tracking pronoun references within a sentence, or parsing syntactic dependencies, are especially vulnerable. The model may still produce reasonable outputs because other mechanisms compensate, but the positional signal that was supposed to guide local attention has become weaker.

Let's visualize this problem:

The compression is dramatic. With Position Interpolation at 8x scale, adjacent tokens are separated by only 0.125 radians in the highest-frequency dimension, compared to 1 radian originally. This means the model has far less "room" to distinguish nearby positions, potentially harming tasks that require precise local attention.

One might ask: why does this matter so much if the model can still distinguish all positions from one another? The answer is that attention is computed via inner products, and inner product differences encode similarity. When adjacent tokens are close together in the rotation space, their inner products with any given query become nearly identical, which means the softmax attention distribution becomes nearly uniform over nearby tokens. The model loses the sharp, precise attention to individual nearby tokens that it learned during training. This loss shows up in perplexity benchmarks and in downstream tasks that require local syntactic sensitivity.