NTK-aware Scaling › NTK-aware Scaling

Position Interpolation, as we saw in the previous chapter, extends context length by simply scaling down all rotation frequencies. While this works, it comes with a hidden cost: by compressing all frequencies equally, we lose the fine-grained positional distinctions that high-frequency components provide. Consider a model trained with 4,096 tokens. Position Interpolation at 8x scale compresses everything to fit 32,768 positions, but now adjacent tokens that were once easily distinguishable become nearly identical in their high-frequency dimensions.

NTK-aware scaling takes a more surgical approach. Instead of treating all frequencies equally, it recognizes that different frequency bands serve different purposes. High frequencies distinguish nearby tokens, while low frequencies capture long-range structure. By scaling frequencies non-uniformly, NTK-aware methods preserve local precision while still extending the effective context window.

Think of it like a camera lens. If you want to photograph a larger scene, you can zoom out uniformly, but then the small details you cared about become blurry. A smarter approach would be to change only the focal length for objects far away while keeping close-up clarity intact. NTK-aware scaling applies exactly this logic to the rotary frequency spectrum, compressing the "distant" low-frequency dimensions while leaving the "close-up" high-frequency ones at full resolution.

The motivation for this approach came from an observation about how neural networks use their positional information. During training, models learn to use different frequency bands for different purposes. High-frequency dimensions encode fine-grained local relationships, things like "this word is immediately followed by that word" or "this is a possessive construction." Low-frequency dimensions encode structural and global relationships, things like "this sentence is in the first half of the document" or "this is part of the concluding paragraph." Position Interpolation disrupts both kinds of encoding equally, even though they have very different sensitivity requirements.

The mathematical insight underlying NTK-aware scaling is elegant: rather than modifying the output frequencies directly, we can achieve dimension-dependent scaling by modifying the base constant of the RoPE formula. Because frequencies decrease exponentially with dimension index, a change to the base has a larger effect on low-frequency dimensions than on high-frequency ones. This gives us the non-uniform scaling we want almost for free, as a mathematical consequence of the exponential structure.

This chapter develops the intuition behind NTK-aware scaling, derives the mathematical formula from first principles, and implements both static and dynamic variants. You will see exactly why this approach often outperforms linear interpolation, and you will leave with a concrete understanding of when to use static versus dynamic scaling in production systems.