Position Interpolation › Limitations of Position Interpolation

Position Interpolation enables longer context, but it comes with trade-offs. Understanding these limitations helps explain why subsequent methods like NTK-aware scaling were developed. This is not a criticism of Position Interpolation; every technique has a regime where it works well and a regime where it struggles. Knowing the boundaries helps you choose the right tool for each situation.

Reduced positional resolution. The most significant limitation is reduced angular resolution between nearby positions. When we compress 8,192 positions into the range of 2,048, each position step produces 1/4 the rotation of the original. Two tokens that are adjacent in the extended sequence differ by the same rotation as tokens 0.25 positions apart in the original. This compression can make it harder for the model to distinguish nearby positions, potentially affecting tasks requiring fine-grained positional awareness. Tasks like coreference resolution (determining which pronoun refers to which noun), syntactic dependency parsing, and named entity recognition all benefit from accurate local position signals. These may degrade with aggressive context extension via Position Interpolation.

Non-uniform frequency scaling. Position Interpolation applies the same scale factor to all frequency components. However, different frequencies may require different treatment. High-frequency components (fast-rotating dimensions) are most affected by the reduced resolution because they distinguish local positions. Low-frequency components (slow-rotating dimensions) were already coarse and are less impacted. This uniform scaling is suboptimal, which motivated the development of NTK-aware scaling that treats frequencies differently.

The key intuition about frequency-selective treatment is this: the low-frequency dimensions have wavelengths that already exceed the training context length. They were not being used to distinguish individual positions at all; they were encoding coarse global structure. Reducing their frequency further does not materially change their behavior. The high-frequency dimensions, by contrast, were the primary source of precise positional information. Halving or quartering their frequency degrades exactly the information the model relied on most. A smarter approach would apply no scaling (or minimal scaling) to high-frequency dimensions and apply more scaling to low-frequency dimensions. This is the idea behind NTK-aware scaling, which we will explore in the next chapter.

Fine-tuning requirement. While Position Interpolation requires far less fine-tuning than training from scratch, it still requires some adaptation. This limits scenarios where you need to extend context on the fly without access to fine-tuning data or compute. Some applications, particularly in deployment environments with strict latency and cost constraints, cannot afford even a 1,000-step fine-tuning run. For these use cases, methods that can extend context without any fine-tuning are more practical.

Perplexity increase. Even after fine-tuning, models with Position Interpolation often show slightly higher perplexity compared to models trained directly on longer sequences. The compression introduces information loss that fine-tuning can mitigate but not fully eliminate. The perplexity gap widens with larger extension factors. A 2x extension causes minimal perplexity increase. A 4x extension causes a modest increase that fine-tuning largely recovers. Beyond 8x, the perplexity gap can become significant, particularly for text types that rely on fine-grained positional cues.

Recency bias disruption. Many language models develop a recency bias: they attend more strongly to nearby tokens than to distant ones. This bias is often useful because nearby context is usually more relevant. Position Interpolation disrupts this bias in a specific way: the effective "distance" between nearby tokens is reduced by the scale factor. Under 4x compression, two adjacent tokens feel like they are 0.25 positions apart rather than 1 position apart. The model's recency bias may weaken as a result, because the gradient between "near" and "far" tokens becomes less steep. Whether this is harmful depends on the task. For tasks where distant context is important (like question answering over long documents), a flatter recency bias may be helpful.

Let's quantify how different dimension pairs are affected by the uniform scaling.

The analysis reveals the asymmetric impact. High-frequency dimension pairs, which originally completed a rotation every 6-50 positions, now require 4x as many positions to complete the same rotation. This directly impairs local position discrimination: the model can no longer tell apart tokens that are 1, 2, or 3 positions away from each other as accurately as it could before. Low-frequency dimension pairs, already operating at wavelengths of thousands of positions, remain capable of distinguishing positions at the extended range, because their wavelengths are still shorter than the new context length.

This analysis is not just theoretical. Empirical results from the original Position Interpolation paper and from subsequent work show that models extended with 4x interpolation perform comparably to their original 2K versions on most benchmarks, with the main degradation appearing on tasks where local syntactic precision matters. The asymmetric frequency impact explains why.

The visualization shows how Position Interpolation uniformly scales all frequencies by the same factor (4x reduction). The vertical lines connecting original (circles) to scaled (squares) values have the same proportional length across all dimension pairs. This uniform treatment is the core limitation that NTK-aware scaling addresses.

From a signal processing perspective, what Position Interpolation does to the frequency spectrum is equivalent to applying a low-pass filter. All frequencies are attenuated by the same multiplicative factor, which is precisely what a pure scaling operation does in the frequency domain. Low-pass filtering preserves coarse structure but loses fine detail, which maps directly onto the observed behavior: long-range dependencies are handled well, but short-range positional precision suffers. NTK-aware scaling, by contrast, applies frequency-selective attenuation, preserving the high-frequency content and attenuating only the frequencies that need adjustment to accommodate the longer context.