NTK-aware Scaling › Comparing NTK to Position Interpolation

Let's directly compare how well each method preserves the relative position encoding properties. The key metric here is how closely the scaled methods track the original RoPE behavior as a function of relative position offset. A method that perfectly preserved all properties would have zero deviation everywhere:

The deviation plot (right) makes the advantage clear: NTK-aware scaling maintains smaller deviations from the original behavior, particularly for small relative positions where local attention patterns matter most.

The deviation for NTK-aware scaling starts near zero at relative position 0 and grows slowly, while Position Interpolation's deviation is larger throughout. This quantifies the intuition we built earlier: NTK-aware scaling is more faithful to the original model's behavior for nearby tokens, which is exactly what we need for precise local attention.

One way to think about this: Position Interpolation is like translating a document with Google Translate and then running spell-check. The translation changes every word, and the spell-checker doesn't know which original word each new word was supposed to correspond to. NTK-aware scaling is like translating only the technical jargon while leaving everyday words unchanged, a more targeted intervention that preserves more of the original meaning.