Position Interpolation › The Extrapolation Problem

Before we can appreciate the solution, we need to understand the failure mode in precise terms. The extrapolation problem is not just that "big numbers are bad." It arises from a specific mismatch between what the model learned and what it is asked to do, and the details of that mismatch matter for understanding why Position Interpolation works.

RoPE encodes position through rotation. At position $m$, each dimension pair of the query and key vectors is rotated by an angle proportional to $m$. The key idea is that each dimension pair rotates at a different frequency, creating a unique positional signature. Think of it as a clock with many hands, each ticking at a different speed. The pattern formed by the positions of all the hands at a given moment encodes the current "time" (position). Two different positions will have two different hand configurations. The rotation angle for dimension pair $i$ at position $m$ is computed as:

where:

• $\theta_i(m)$: the rotation angle (in radians) applied to dimension pair $i$ at sequence position $m$
• $m$: the position index in the sequence (0, 1, 2, ..., $L-1$ for a sequence of length $L$)
• $i$: the dimension pair index (0, 1, 2, ..., $d/2 - 1$)
• $d$: the total embedding dimension (typically 64, 128, or larger)
• $\text{base}$: a constant that controls the range of frequencies (typically 10000)
• $\theta_i = 1/\text{base}^{2i/d}$: the base frequency for dimension pair $i$, which decreases exponentially as $i$ increases

The exponential decay in the base frequency $\theta_i$ means that dimension pair 0 rotates fastest (with $\theta_0 = 1$ radian per position), while higher-indexed pairs rotate progressively slower. This creates a multi-scale representation where different dimension pairs capture positional information at different granularities. The fastest dimension pairs complete a full rotation every few positions, making them sensitive to fine-grained local position distinctions. The slowest dimension pairs complete a rotation only after tens of thousands of positions, making them sensitive to coarse global structure. Together, they form a rich positional encoding that the transformer can read like a coordinate system.

Notice that the angular resolution of this coordinate system is calibrated to the training sequence length. The frequencies were chosen (using the base of 10,000) so that the full spectrum of rotation patterns fits comfortably within the context window. When you push beyond that window, you are not just asking the model to handle new numbers. You are asking it to interpret rotation patterns that were never part of the coordinate system it was trained to read.

During training on sequences of length $L$, the model sees positions $0, 1, 2, \ldots, L-1$. The rotation angles range from 0 to $(L-1) \cdot \theta_i$ for each dimension pair. Every pattern the model has learned to interpret lies somewhere in this bounded space.

What happens when we present the model with position $2L$? The rotation angle $2L \cdot \theta_i$ exceeds anything seen during training. For the fastest-rotating dimensions (where $\theta_i$ is large), these new angles produce embedding rotations the attention mechanism has never learned to interpret. In practice, this means that when a token at position 3,000 tries to attend to a token at position 100, the relative rotation computed from those positions falls entirely outside what the model learned during training. The attention score becomes meaningless, not because the model cannot compute a number, but because the number it computes carries no learned meaning.

Let's visualize this problem by examining the rotation angles at different positions.

For the fastest-rotating dimension pair (pair 0), the model sees angles up to about 2,047 radians during training. Extending to 4x the context pushes this to over 8,000 radians. While both values wrap around the unit circle many times, the relative patterns between dimension pairs change in ways the model hasn't learned.

The real problem becomes clear when we examine what happens to attention patterns. During training, the model learns that certain rotation angle combinations correspond to meaningful relative positions. When extrapolated angles produce unfamiliar combinations, the learned attention patterns break down.

The plot reveals the core issue. Fast-rotating dimensions (low indices) experience dramatic angle increases during extrapolation. The angle at position 8191 for dimension pair 0 is four times larger than at position 2047. Meanwhile, slow-rotating dimensions (high indices) barely change. This asymmetric behavior disrupts the carefully balanced patterns the model learned during training.

The asymmetry matters for an important reason. The fast-rotating dimensions are responsible for distinguishing nearby positions, the very cases where the model most needs accurate positional information. When a model must determine whether a pronoun refers to the noun in the immediately preceding clause versus the one three sentences earlier, it relies on high-frequency positional signals. Those are exactly the signals that break under extrapolation. The slow-rotating dimensions, which encode coarse global position, remain more stable but are not sufficient on their own.