Position Interpolation

Modern language models face a fundamental tension: training on long sequences is expensive, but real-world applications demand them. A model trained on 2,048 tokens might encounter documents with 8,000 tokens, conversations spanning 16,000 tokens, or codebases requiring even longer context. When RoPE-based models try to process positions beyond their training range, performance degrades rapidly. The rotation angles reach values the model has never seen, producing attention patterns that bear no resemblance to what was learned.

To appreciate why this is such a thorny problem, recall how Rotary Position Embeddings work. RoPE encodes position information directly into the query and key vectors by rotating them. Each dimension pair in the embedding vector rotates at a characteristic frequency, and the interaction between two positions during attention is determined by the relative rotation between those dimension pairs. The model learns, during pretraining, which relative rotation patterns correspond to meaningful positional relationships. Positions that are two tokens apart produce one rotation pattern; positions that are fifty tokens apart produce another. These learned associations are what allow the model to understand that "the dog chased the cat" has a different meaning than "the cat chased the dog," even when both sentences contain the same words.

The problem is that this learned mapping is bounded. A model trained on sequences up to 2,048 tokens never encounters position 3,000, or 5,000, or 8,000. The rotation angles at those positions fall entirely outside the training distribution. The model has no principled way to interpret them. In the best case, performance degrades gracefully. In the worst case, the attention mechanism produces completely incoherent patterns, essentially treating every token as equally relevant (or irrelevant) to every other token regardless of distance.

Position Interpolation, introduced by Chen et al. in 2023, offers an elegant solution: instead of extrapolating to unseen positions, we interpolate within the familiar range. Rather than assigning position 4,096 a rotation angle the model has never encountered, we scale all positions down so they fit within the original training range. Position 4,096 becomes position 2,048 after scaling, and the model sees familiar rotation angles even as it processes longer sequences.

The key insight is that interpolation is a much more forgiving operation for neural networks than extrapolation. When you ask a model to evaluate a rotation angle it has seen during training, it can use everything it learned. When you ask it to evaluate a rotation angle it has never seen, it must generalize beyond its experience. Neural networks are notoriously good at the former and notoriously unreliable at the latter. Position Interpolation trades one well-defined problem (out-of-distribution angles) for a different, more tractable one (compressed angular resolution).

This chapter develops Position Interpolation from first principles. We'll start by understanding why RoPE fails at extrapolation, derive the interpolation formula, implement it in code, and explore its limitations. By the end, you'll understand both the elegance of this approach and why subsequent methods like NTK-aware scaling were developed to address its shortcomings.

Before we can appreciate the solution, we need to understand the failure mode in precise terms. The extrapolation problem is not just that "big numbers are bad." It arises from a specific mismatch between what the model learned and what it is asked to do, and the details of that mismatch matter for understanding why Position Interpolation works.

RoPE encodes position through rotation. At position $m$, each dimension pair of the query and key vectors is rotated by an angle proportional to $m$. The key idea is that each dimension pair rotates at a different frequency, creating a unique positional signature. Think of it as a clock with many hands, each ticking at a different speed. The pattern formed by the positions of all the hands at a given moment encodes the current "time" (position). Two different positions will have two different hand configurations. The rotation angle for dimension pair $i$ at position $m$ is computed as:

where:

• $\theta_i(m)$: the rotation angle (in radians) applied to dimension pair $i$ at sequence position $m$
• $m$: the position index in the sequence (0, 1, 2, ..., $L-1$ for a sequence of length $L$)
• $i$: the dimension pair index (0, 1, 2, ..., $d/2 - 1$)
• $d$: the total embedding dimension (typically 64, 128, or larger)
• $\text{base}$: a constant that controls the range of frequencies (typically 10000)
• $\theta_i = 1/\text{base}^{2i/d}$: the base frequency for dimension pair $i$, which decreases exponentially as $i$ increases

The exponential decay in the base frequency $\theta_i$ means that dimension pair 0 rotates fastest (with $\theta_0 = 1$ radian per position), while higher-indexed pairs rotate progressively slower. This creates a multi-scale representation where different dimension pairs capture positional information at different granularities. The fastest dimension pairs complete a full rotation every few positions, making them sensitive to fine-grained local position distinctions. The slowest dimension pairs complete a rotation only after tens of thousands of positions, making them sensitive to coarse global structure. Together, they form a rich positional encoding that the transformer can read like a coordinate system.

Notice that the angular resolution of this coordinate system is calibrated to the training sequence length. The frequencies were chosen (using the base of 10,000) so that the full spectrum of rotation patterns fits comfortably within the context window. When you push beyond that window, you are not just asking the model to handle new numbers. You are asking it to interpret rotation patterns that were never part of the coordinate system it was trained to read.

The plot reveals the core issue. Fast-rotating dimensions (low indices) experience dramatic angle increases during extrapolation. The angle at position 8191 for dimension pair 0 is four times larger than at position 2047. Meanwhile, slow-rotating dimensions (high indices) barely change. This asymmetric behavior disrupts the carefully balanced patterns the model learned during training.

The asymmetry matters for an important reason. The fast-rotating dimensions are responsible for distinguishing nearby positions, the very cases where the model most needs accurate positional information. When a model must determine whether a pronoun refers to the noun in the immediately preceding clause versus the one three sentences earlier, it relies on high-frequency positional signals. Those are exactly the signals that break under extrapolation. The slow-rotating dimensions, which encode coarse global position, remain more stable but are not sufficient on their own.

We've seen that extrapolation fails because the model encounters rotation angles outside its training distribution. But what if we could ensure that every position, no matter how far into the extended sequence, produces angles the model has already seen? This is the core intuition behind Position Interpolation.

The key insight is deceptively simple: the model does not care about absolute positions. It cares about rotation patterns. If we can arrange for every position in the extended sequence to produce a rotation pattern that falls within the training distribution, the model can apply what it learned even on longer sequences. We achieve this by scaling positions down rather than letting them grow without bound.

Think of the training range as a ruler with marks at integer positions from 0 to 2,047. During training, the model learned to interpret positions at every mark on this ruler. Now we have a longer sequence with 8,192 tokens. We need to assign each token a position, and we want every position to fall on the original ruler. The solution is not to build a longer ruler; it is to compress the new tokens so they all fit within the existing ruler's range. We can imagine each position being lightly etched between the existing marks, creating a finer-grained version of the original scale. The marks are denser, but they all fall within the same range the model already knows.

If we want to process a sequence of length $L_{\text{target}}$ where $L_{\text{target}} > L_{\text{train}}$, we define a scale factor that maps the extended range back to the familiar one:

where:

• $s$: the scale factor, always between 0 and 1 when extending context
• $L_{\text{train}}$: the maximum sequence length seen during training (e.g., 2048)
• $L_{\text{target}}$: the target extended sequence length (e.g., 8192)

This scale factor answers the question: "How much do we need to shrink the extended positions to fit them within the training range?" For a 4x extension (2K to 8K), $s = 2048/8192 = 0.25$, meaning every position is compressed to one-quarter of its original value.

Let's trace through concrete examples to see how this mapping works. For a model trained on 2,048 positions processing an 8,192-token sequence:

Every position in the extended sequence, no matter how large, maps to a value the model encountered during training. Position 8,191 in the extended sequence produces the same rotation pattern as position 2,047 would in the original system. The endpoint of the 8K sequence is treated the same as the endpoint of the 2K training window. Everything in between is compressed proportionally.

In practice, this means that neighboring tokens in a very long context window are closer together in "position space" than they would be in a shorter sequence. The model must learn, through fine-tuning, that a small angular difference now represents a larger actual distance. This is a much easier learning problem than generalizing to entirely unseen rotation patterns. You are adjusting the calibration of a known instrument rather than inventing a new one from scratch.

Now we can formalize this intuition mathematically. The derivation proceeds in three steps, each building naturally on the previous. The goal is to arrive at a modified formula that takes an actual position $m$ in the extended sequence and returns a rotation angle that falls within the range the model saw during training.

Step 1: Recall the original RoPE rotation angle.

In standard RoPE, the rotation angle for dimension pair $i$ at position $m$ is simply the product of position and base frequency:

where $\theta_i = 1/\text{base}^{2i/d}$ is the base frequency for dimension pair $i$. This formula produces angles that grow linearly with position, which is exactly the behavior that causes extrapolation to fail when $m$ exceeds the training range.

Step 2: Apply position scaling to compress the range.

Position Interpolation modifies this formula by scaling the position index before computing the rotation. Instead of using $m$ directly, we use $m \cdot s$:

We can rearrange this expression to reveal an important insight:

where:

• $\theta_i^{\text{PI}}(m)$: the position-interpolated rotation angle for dimension pair $i$ at position $m$
• $m$: the actual position in the extended sequence
• $s$: the scale factor ($L_{\text{train}} / L_{\text{target}}$)
• $\theta_i$: the original base frequency for dimension pair $i$

The rearrangement shows two equivalent interpretations of Position Interpolation:

1. Scale the position: Compute angles for position $m \cdot s$ using original frequencies
2. Scale the frequency: Compute angles for position $m$ using reduced frequencies $s \cdot \theta_i$

Both perspectives lead to the same result, but the second interpretation proves useful for implementation.

Step 3: Express as modified base frequency.

Let's push the algebraic manipulation further to see what Position Interpolation does to the effective RoPE base:

We can factor out the scale to express this in terms of a modified base:

This reveals that Position Interpolation is mathematically equivalent to using a larger effective base:

For extending from 2K to 8K context ($s = 0.25$), the effective base becomes $\text{base}' = 10000 / 0.25 = 40000$. Why does a larger base help? Recall that the base frequency is $\theta_i = 1/\text{base}^{2i/d}$. A larger base produces smaller frequencies, which means slower rotations for all dimension pairs. Slower rotations let us fit more positions into the same angular range before exceeding the training maximum.

The mathematical derivation reveals something elegant: Position Interpolation doesn't change the fundamental structure of RoPE. It doesn't add new components or modify the attention mechanism. It simply asks: "What if we used a different base constant from the start?" The answer is that a larger base would have allowed longer sequences all along, but at the cost of reduced angular resolution between nearby positions.

This insight also explains why fine-tuning is necessary. The model learned to associate certain rotation patterns with certain relative distances. When we compress positions, those associations break. A distance of 100 positions now produces the same rotation pattern as 25 positions would have during training. Fine-tuning recalibrates these associations.

Notice that this is a rather gentle disruption. The model already knows how to process relative distances of 0 to 2,047. After Position Interpolation, a distance of 8,191 compresses to 2,047. A distance of 4,096 compresses to 1,024. The relative ordering of all positions is preserved. No two distinct positions are mapped to the same effective position (the mapping is injective). The only change is that the "units" of the positional coordinate system have been rescaled by a factor of four. Fine-tuning teaches the model that the rescaled units are still meaningful, just on a different scale.

Think of it like switching from measuring distances in meters to measuring them in kilometers. All the same relationships hold; the numbers are just four times smaller. A person who knows that 100 meters is a short sprint can quickly adapt to understanding that 0.1 kilometers is also a short sprint. The positional geometry is preserved; only the units change.

Before writing code, let's work through a concrete numerical example to solidify the concept. This will make the subsequent implementation much easier to follow.

Suppose we have a model with $d_{\text{model}} = 4$ (two dimension pairs, which is unrealistically small but easy to trace), $\text{base} = 10000$, trained on sequences up to $L_{\text{train}} = 8$ positions. We now want to process a sequence of $L_{\text{target}} = 16$ positions.

The scale factor is:

The base frequencies for our two dimension pairs ($i = 0$ and $i = 1$ with $d = 4$) are:

Now consider position $m = 12$ in the extended sequence. Under standard RoPE (no interpolation), the rotation angles would be:

• Dimension pair 0: $12 \times 1.0 = 12.0$ radians
• Dimension pair 1: $12 \times 0.01 = 0.12$ radians

The training maximum (at position 7) was:

• Dimension pair 0: $7 \times 1.0 = 7.0$ radians
• Dimension pair 1: $7 \times 0.01 = 0.07$ radians

Position 12 exceeds the training maximum for both dimension pairs. The fast-rotating dimension pair 0 is especially far outside the training range (12.0 vs 7.0 radians).

Under Position Interpolation, we scale the position first: $m' = 12 \times 0.5 = 6.0$. The rotation angles become:

• Dimension pair 0: $6.0 \times 1.0 = 6.0$ radians (within training max of 7.0)
• Dimension pair 1: $6.0 \times 0.01 = 0.06$ radians (within training max of 0.07)

Both angles now fall within the range the model saw during training. The model can apply its learned interpretation of these rotation patterns to process what is effectively a mid-sequence position. The trade-off is that position 12 in the extended sequence "feels like" position 6 in the original sequence, meaning the model perceives it as occupying roughly the middle of the context rather than three-quarters of the way through. Fine-tuning corrects this calibration.

With the mathematical foundation in place, let's translate Position Interpolation into code. The implementation is remarkably simple: we compute the scale factor, multiply positions by that factor, and then proceed with standard RoPE angle computation.

The core function takes positions along with both the training and target lengths. It computes the scale factor internally, scales all positions, and returns angles that stay within the training range.

The critical test: do the interpolated angles at the maximum extended position match the training maximum? Let's compare the angles at position 8191 across three methods.

The heatmap makes the difference visually striking. The top row shows the intense "heat" of original RoPE's extrapolated angles, especially in the fast-rotating dimensions on the left. Position Interpolation (middle row) shows the same pattern as the training maximum (bottom row), confirming that we've successfully mapped extended positions back into familiar territory.

Let's visualize this compression graphically. The plot below shows how actual positions in the extended sequence map to effective positions for RoPE computation.

Why does interpolation work better than extrapolation? The answer lies in how neural networks generalize. During training, the model learns attention patterns for rotation angles in a specific range. These learned patterns form a continuous function over that range.

When we extrapolate, we ask the model to generalize this function to inputs it has never seen. Neural networks are notoriously poor at extrapolation; they often produce arbitrary outputs outside their training distribution. This is not a failure of the specific model; it is a fundamental property of function approximators trained with finite data. The model's learned function is essentially undefined outside the convex hull of its training inputs, and there is no reason to expect it to behave sensibly there.

When we interpolate, we stay within the training distribution but query it at finer-grained positions. The model can use its learned continuous representations to handle intermediate values. This works because neural network functions are typically smooth within the training region. They do not suddenly develop sharp discontinuities or erratic behavior between two neighboring training points. An interpolated position between two training positions is handled gracefully by the smooth learned function.

Consider an analogy. Imagine training someone to recognize temperatures between 0°C and 100°C, using only integer values. If you then ask them about 200°C, they must extrapolate beyond their experience, and their predictions become unreliable. They have no basis for judgment. But if you ask about 37.5°C, they can interpolate from their knowledge of 37°C and 38°C. The intermediate value lies in a region they understand, and their estimate will be close to correct because temperature perception is continuous.

The same principle applies to rotation angles. The model's attention mechanism is a smooth function of rotation angles within the training range. Interpolated positions produce rotation angles that lie between training values, and the model's smooth learned function handles them well. Extrapolated positions produce angles beyond the training range, where the function is unpredictable.

Let's quantify this by examining how the angle differences (which determine attention scores) change under interpolation.

Position Interpolation alone doesn't magically enable long context. While the rotation angles stay within the training distribution, the model still encounters unfamiliar situations. Two tokens that were 100 positions apart during training now produce the same relative rotation as tokens 400 positions apart in the extended sequence. The model must learn to interpret these compressed position signals.

This recalibration is the heart of what fine-tuning accomplishes. Before fine-tuning, the model has a deeply ingrained prior: a rotation difference of $\Delta\theta$ means that the two tokens are a certain distance apart. That prior was learned from billions of training examples where a relative rotation of, say, 0.25 radians in dimension pair 0 always indicated that the tokens were separated by about one position. After Position Interpolation, that same rotation difference now indicates a separation of about four positions. The model's prior is wrong, and without correction, it produces suboptimal attention patterns.

This is where fine-tuning comes in. After applying Position Interpolation, models typically undergo a short fine-tuning phase on long-context data. The good news: this fine-tuning is remarkably efficient. Chen et al. found that only about 1,000 fine-tuning steps were needed to adapt a 2K-context LLaMA model to 8K context, compared to the billions of tokens used in original pretraining.

Why does the fine-tuning converge so quickly? Several factors contribute. First, the basic capability to model language, understand syntax, and reason about text is entirely preserved. The model does not need to relearn how language works; it just needs to relearn the scale of its positional coordinate system. Second, all rotation angles remain within the training distribution, meaning the model's existing representations are never completely invalidated. They are being recalibrated, not rebuilt. Third, the mapping from old positions to new positions is simple and consistent: a uniform linear scaling. Once the model grasps this scaling, it can generalize across all positions in the extended sequence.

In practice, long-context fine-tuning data does not need to be task-specific. A diverse mix of long documents, ranging from books and articles to code and conversations, is sufficient for the model to recalibrate its positional understanding. The training objective can be the same standard next-token prediction used in pretraining.

Let's simulate what fine-tuning might need to correct by examining how attention patterns change under interpolation.

The plots illustrate the core transformation. With Position Interpolation, a distance of 4,096 positions produces the same rotation angle pattern as a distance of 1,024 positions in standard RoPE. Fine-tuning teaches the model that this compressed pattern now represents the longer distance.

The second plot also reveals something subtle: at very short distances (the first few hundred positions), the interpolated angle variance closely tracks the standard RoPE curve. This is because short distances already produced small angles in standard RoPE, and compressing them by 4x still leaves them small. The model's short-range attention patterns are least disrupted by Position Interpolation. The most significant disruption occurs at distances approaching the training context limit, where standard RoPE produces its largest relative angles and Position Interpolation compresses them the most.

Let's put everything together into a complete Position Interpolation implementation that can be applied to RoPE. The class below combines all the pieces we've developed: scale factor computation, position scaling, frequency precomputation, and rotation matrix construction. Understanding this implementation end-to-end is valuable because production implementations in frameworks like Hugging Face Transformers follow exactly the same logic, just with additional optimizations for batched computation on GPU hardware.

The interpolated rotation matrix at position 6,000 matches the standard rotation matrix at position 1,500, confirming that our implementation correctly maps extended positions back to the training range.

The "Training Equiv" column in the output is computed by scaling position 6,000 by 0.25 to get 1,500, then applying standard RoPE. The "Interpolated" column applies Position Interpolation directly. They should be identical up to floating-point precision. This equivalence provides a sanity check and the mathematical guarantee that Position Interpolation delivers: every position in the extended sequence is handled exactly as if it were a proportionally smaller position in the original system.

Position Interpolation enables longer context, but it comes with trade-offs. Understanding these limitations helps explain why subsequent methods like NTK-aware scaling were developed. This is not a criticism of Position Interpolation; every technique has a regime where it works well and a regime where it struggles. Knowing the boundaries helps you choose the right tool for each situation.

Reduced positional resolution. The most significant limitation is reduced angular resolution between nearby positions. When we compress 8,192 positions into the range of 2,048, each position step produces 1/4 the rotation of the original. Two tokens that are adjacent in the extended sequence differ by the same rotation as tokens 0.25 positions apart in the original. This compression can make it harder for the model to distinguish nearby positions, potentially affecting tasks requiring fine-grained positional awareness. Tasks like coreference resolution (determining which pronoun refers to which noun), syntactic dependency parsing, and named entity recognition all benefit from accurate local position signals. These may degrade with aggressive context extension via Position Interpolation.

The visualization shows how Position Interpolation uniformly scales all frequencies by the same factor (4x reduction). The vertical lines connecting original (circles) to scaled (squares) values have the same proportional length across all dimension pairs. This uniform treatment is the core limitation that NTK-aware scaling addresses.

From a signal processing perspective, what Position Interpolation does to the frequency spectrum is equivalent to applying a low-pass filter. All frequencies are attenuated by the same multiplicative factor, which is precisely what a pure scaling operation does in the frequency domain. Low-pass filtering preserves coarse structure but loses fine detail, which maps directly onto the observed behavior: long-range dependencies are handled well, but short-range positional precision suffers. NTK-aware scaling, by contrast, applies frequency-selective attenuation, preserving the high-frequency content and attenuating only the frequencies that need adjustment to accommodate the longer context.

When implementing Position Interpolation, the following parameters control the behavior:

• train_length: The maximum sequence length the model was originally trained on (e.g., 2048 for LLaMA). This defines the upper bound of positions the model has learned to interpret during pretraining. It is typically documented in the model's configuration file and is not something you change; it is a property of the pretrained model.

• target_length: The extended context length you want to support (e.g., 8192). Must be greater than train_length for interpolation to apply. Choosing this value requires thinking about your actual use case: what is the longest document or conversation you expect to encounter? Setting this too large degrades performance at all lengths; setting it too small may leave some inputs truncated.

• scale: Computed as train_length / target_length. This determines how much to compress positions. A scale of 0.25 means 4x context extension. Smaller scale values enable longer contexts but reduce positional resolution proportionally. You do not set this directly; it is derived from the other two parameters.

• base: The RoPE base constant (typically 10000). Position Interpolation effectively increases this to base / scale, slowing all rotations uniformly. You normally leave this at the model's default; it only matters if you are experimenting with modified base values.

• d_model: The embedding dimension. Must be even since RoPE operates on dimension pairs. Each pair rotates at a different frequency determined by its index. This is fixed by the model architecture and cannot be changed during inference.

When selecting parameters, consider the trade-off between context length and positional resolution. Extending by 2x is generally safe with minimal fine-tuning. Extensions of 4x-8x work well but require more fine-tuning to recover performance. Extensions beyond 8x may significantly degrade the model's ability to distinguish nearby positions. There is also a practical memory consideration: quadrupling the context length requires roughly quadrupling the memory for the KV cache, assuming standard multi-head attention. Techniques like sliding-window attention or sparse attention can help here, but they are orthogonal to the position encoding question.

Position Interpolation provides an elegant solution to the context length extension problem. By scaling position indices rather than extrapolating to unseen values, it keeps all rotation angles within the training distribution. The key insights include:

• Interpolation over extrapolation. Neural networks generalize poorly outside their training distribution. By scaling positions down instead of letting them grow, Position Interpolation stays within familiar territory. The model never sees a rotation angle it wasn't trained to interpret.

• Simple implementation. The technique requires only a scale factor applied to position indices before computing RoPE. No architectural changes are needed. In terms of code, it is literally one multiplication.

• Equivalent to base scaling. Position Interpolation is mathematically equivalent to using a larger RoPE base. A 4x extension from 2K to 8K is the same as using a base of 40,000 instead of 10,000. This equivalence is useful because it connects Position Interpolation to the broader family of scaled-base RoPE methods.

• Efficient fine-tuning. Adapting a model to extended context requires only about 1,000 fine-tuning steps, orders of magnitude less than original pretraining. This efficiency comes from the fact that the model's general language understanding is preserved; only the positional calibration needs adjustment.

• Uniform scaling limitation. All frequency components receive the same scaling treatment, which is suboptimal. High-frequency dimensions, which carry local position information, lose the most resolution. This is the most significant practical limitation and the primary motivation for subsequent methods.

Position Interpolation demonstrated that context length is not a fixed property of a pretrained model but rather a parameter that can be extended through lightweight adaptation. This insight was foundational for the wave of long-context models that appeared in 2023 and 2024. It showed that a model trained on 2K tokens could, with minimal effort, process 8K tokens reliably, and that this capability had been latent in the model all along, waiting for the right positional scaling to make it usable.

The limitations of Position Interpolation, particularly its uniform treatment of frequencies, motivated the development of NTK-aware scaling, which we'll explore in the next chapter. That technique applies frequency-dependent scaling, preserving high-frequency components while adjusting low-frequency ones, achieving better performance without sacrificing local positional awareness. Understanding why Position Interpolation falls short at high extension factors is precisely what makes NTK-aware scaling's design choices intuitive.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Position Interpolation and extending context length in language models.