Position Interpolation › Position Interpolation

Modern language models face a fundamental tension: training on long sequences is expensive, but real-world applications demand them. A model trained on 2,048 tokens might encounter documents with 8,000 tokens, conversations spanning 16,000 tokens, or codebases requiring even longer context. When RoPE-based models try to process positions beyond their training range, performance degrades rapidly. The rotation angles reach values the model has never seen, producing attention patterns that bear no resemblance to what was learned.

To appreciate why this is such a thorny problem, recall how Rotary Position Embeddings work. RoPE encodes position information directly into the query and key vectors by rotating them. Each dimension pair in the embedding vector rotates at a characteristic frequency, and the interaction between two positions during attention is determined by the relative rotation between those dimension pairs. The model learns, during pretraining, which relative rotation patterns correspond to meaningful positional relationships. Positions that are two tokens apart produce one rotation pattern; positions that are fifty tokens apart produce another. These learned associations are what allow the model to understand that "the dog chased the cat" has a different meaning than "the cat chased the dog," even when both sentences contain the same words.

The problem is that this learned mapping is bounded. A model trained on sequences up to 2,048 tokens never encounters position 3,000, or 5,000, or 8,000. The rotation angles at those positions fall entirely outside the training distribution. The model has no principled way to interpret them. In the best case, performance degrades gracefully. In the worst case, the attention mechanism produces completely incoherent patterns, essentially treating every token as equally relevant (or irrelevant) to every other token regardless of distance.

Position Interpolation, introduced by Chen et al. in 2023, offers an elegant solution: instead of extrapolating to unseen positions, we interpolate within the familiar range. Rather than assigning position 4,096 a rotation angle the model has never encountered, we scale all positions down so they fit within the original training range. Position 4,096 becomes position 2,048 after scaling, and the model sees familiar rotation angles even as it processes longer sequences.

The key insight is that interpolation is a much more forgiving operation for neural networks than extrapolation. When you ask a model to evaluate a rotation angle it has seen during training, it can use everything it learned. When you ask it to evaluate a rotation angle it has never seen, it must generalize beyond its experience. Neural networks are notoriously good at the former and notoriously unreliable at the latter. Position Interpolation trades one well-defined problem (out-of-distribution angles) for a different, more tractable one (compressed angular resolution).

This chapter develops Position Interpolation from first principles. We'll start by understanding why RoPE fails at extrapolation, derive the interpolation formula, implement it in code, and explore its limitations. By the end, you'll understand both the elegance of this approach and why subsequent methods like NTK-aware scaling were developed to address its shortcomings.