Position Interpolation › The Interpolation Insight

We've seen that extrapolation fails because the model encounters rotation angles outside its training distribution. But what if we could ensure that every position, no matter how far into the extended sequence, produces angles the model has already seen? This is the core intuition behind Position Interpolation.

The key insight is deceptively simple: the model does not care about absolute positions. It cares about rotation patterns. If we can arrange for every position in the extended sequence to produce a rotation pattern that falls within the training distribution, the model can apply what it learned even on longer sequences. We achieve this by scaling positions down rather than letting them grow without bound.

Think of the training range as a ruler with marks at integer positions from 0 to 2,047. During training, the model learned to interpret positions at every mark on this ruler. Now we have a longer sequence with 8,192 tokens. We need to assign each token a position, and we want every position to fall on the original ruler. The solution is not to build a longer ruler; it is to compress the new tokens so they all fit within the existing ruler's range. We can imagine each position being lightly etched between the existing marks, creating a finer-grained version of the original scale. The marks are denser, but they all fall within the same range the model already knows.

If we want to process a sequence of length $L_{\text{target}}$ where $L_{\text{target}} > L_{\text{train}}$, we define a scale factor that maps the extended range back to the familiar one:

where:

• $s$: the scale factor, always between 0 and 1 when extending context
• $L_{\text{train}}$: the maximum sequence length seen during training (e.g., 2048)
• $L_{\text{target}}$: the target extended sequence length (e.g., 8192)

This scale factor answers the question: "How much do we need to shrink the extended positions to fit them within the training range?" For a 4x extension (2K to 8K), $s = 2048/8192 = 0.25$, meaning every position is compressed to one-quarter of its original value.

Let's trace through concrete examples to see how this mapping works. For a model trained on 2,048 positions processing an 8,192-token sequence:

Every position in the extended sequence, no matter how large, maps to a value the model encountered during training. Position 8,191 in the extended sequence produces the same rotation pattern as position 2,047 would in the original system. The endpoint of the 8K sequence is treated the same as the endpoint of the 2K training window. Everything in between is compressed proportionally.

In practice, this means that neighboring tokens in a very long context window are closer together in "position space" than they would be in a shorter sequence. The model must learn, through fine-tuning, that a small angular difference now represents a larger actual distance. This is a much easier learning problem than generalizing to entirely unseen rotation patterns. You are adjusting the calibration of a known instrument rather than inventing a new one from scratch.

Now we can formalize this intuition mathematically. The derivation proceeds in three steps, each building naturally on the previous. The goal is to arrive at a modified formula that takes an actual position $m$ in the extended sequence and returns a rotation angle that falls within the range the model saw during training.

Step 1: Recall the original RoPE rotation angle.

In standard RoPE, the rotation angle for dimension pair $i$ at position $m$ is simply the product of position and base frequency:

where $\theta_i = 1/\text{base}^{2i/d}$ is the base frequency for dimension pair $i$. This formula produces angles that grow linearly with position, which is exactly the behavior that causes extrapolation to fail when $m$ exceeds the training range.

Step 2: Apply position scaling to compress the range.

Position Interpolation modifies this formula by scaling the position index before computing the rotation. Instead of using $m$ directly, we use $m \cdot s$:

We can rearrange this expression to reveal an important insight:

where:

• $\theta_i^{\text{PI}}(m)$: the position-interpolated rotation angle for dimension pair $i$ at position $m$
• $m$: the actual position in the extended sequence
• $s$: the scale factor ($L_{\text{train}} / L_{\text{target}}$)
• $\theta_i$: the original base frequency for dimension pair $i$

The rearrangement shows two equivalent interpretations of Position Interpolation:

1. Scale the position: Compute angles for position $m \cdot s$ using original frequencies
2. Scale the frequency: Compute angles for position $m$ using reduced frequencies $s \cdot \theta_i$

Both perspectives lead to the same result, but the second interpretation proves useful for implementation.

Step 3: Express as modified base frequency.

Let's push the algebraic manipulation further to see what Position Interpolation does to the effective RoPE base:

We can factor out the scale to express this in terms of a modified base:

This reveals that Position Interpolation is mathematically equivalent to using a larger effective base:

For extending from 2K to 8K context ($s = 0.25$), the effective base becomes $\text{base}' = 10000 / 0.25 = 40000$. Why does a larger base help? Recall that the base frequency is $\theta_i = 1/\text{base}^{2i/d}$. A larger base produces smaller frequencies, which means slower rotations for all dimension pairs. Slower rotations let us fit more positions into the same angular range before exceeding the training maximum.

The mathematical derivation reveals something elegant: Position Interpolation doesn't change the fundamental structure of RoPE. It doesn't add new components or modify the attention mechanism. It simply asks: "What if we used a different base constant from the start?" The answer is that a larger base would have allowed longer sequences all along, but at the cost of reduced angular resolution between nearby positions.

This insight also explains why fine-tuning is necessary. The model learned to associate certain rotation patterns with certain relative distances. When we compress positions, those associations break. A distance of 100 positions now produces the same rotation pattern as 25 positions would have during training. Fine-tuning recalibrates these associations.

Notice that this is a rather gentle disruption. The model already knows how to process relative distances of 0 to 2,047. After Position Interpolation, a distance of 8,191 compresses to 2,047. A distance of 4,096 compresses to 1,024. The relative ordering of all positions is preserved. No two distinct positions are mapped to the same effective position (the mapping is injective). The only change is that the "units" of the positional coordinate system have been rescaled by a factor of four. Fine-tuning teaches the model that the rescaled units are still meaningful, just on a different scale.

Think of it like switching from measuring distances in meters to measuring them in kilometers. All the same relationships hold; the numbers are just four times smaller. A person who knows that 100 meters is a short sprint can quickly adapt to understanding that 0.1 kilometers is also a short sprint. The positional geometry is preserved; only the units change.