Position Interpolation › Interpolation vs Extrapolation: A Closer Look
Language AI HandbookAll chapters Position Interpolation › Interpolation vs Extrapolation: A Closer LookOne section · Sections in this chapter6 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Why does interpolation work better than extrapolation? The answer lies in how neural networks generalize. During training, the model learns attention patterns for rotation angles in a specific range. These learned patterns form a continuous function over that range.
When we extrapolate, we ask the model to generalize this function to inputs it has never seen. Neural networks are notoriously poor at extrapolation; they often produce arbitrary outputs outside their training distribution. This is not a failure of the specific model; it is a fundamental property of function approximators trained with finite data. The model's learned function is essentially undefined outside the convex hull of its training inputs, and there is no reason to expect it to behave sensibly there.
When we interpolate, we stay within the training distribution but query it at finer-grained positions. The model can use its learned continuous representations to handle intermediate values. This works because neural network functions are typically smooth within the training region. They do not suddenly develop sharp discontinuities or erratic behavior between two neighboring training points. An interpolated position between two training positions is handled gracefully by the smooth learned function.
Consider an analogy. Imagine training someone to recognize temperatures between 0°C and 100°C, using only integer values. If you then ask them about 200°C, they must extrapolate beyond their experience, and their predictions become unreliable. They have no basis for judgment. But if you ask about 37.5°C, they can interpolate from their knowledge of 37°C and 38°C. The intermediate value lies in a region they understand, and their estimate will be close to correct because temperature perception is continuous.
The same principle applies to rotation angles. The model's attention mechanism is a smooth function of rotation angles within the training range. Interpolated positions produce rotation angles that lie between training values, and the model's smooth learned function handles them well. Extrapolated positions produce angles beyond the training range, where the function is unpredictable.
Let's quantify this by examining how the angle differences (which determine attention scores) change under interpolation.
def compute_relative_angles(positions, d_model=64, base=10000, scale=1.0):
"""Compute relative rotation angles between consecutive positions.
Args:
positions: Array of position indices
d_model: Embedding dimension
base: Base for frequency computation
scale: Position scale factor (1.0 for standard RoPE)
Returns:
relative_angles: Angle difference between position m and m-1
"""
positions = np.array(positions) * scale
dim_pairs = d_model // 2
i = np.arange(dim_pairs)
theta = 1.0 / (base ** (2 * i / d_model))
# Relative angle is just theta * scale (constant for all positions)
relative_angles = theta * scale
return relative_angles
# Compare relative angles for standard RoPE vs interpolated
standard_relative = compute_relative_angles(extended_positions, scale=1.0)
interpolated_relative = compute_relative_angles(extended_positions, scale=scale)Rotation angle per position step (radians): ------------------------------------------------------- Pair Standard RoPE Position Interpolation ------------------------------------------------------- 0 1.000000 0.250000 1 0.749894 0.187474 2 0.562341 0.140585 15 0.013335 0.003334 31 0.000133 0.000033
The relative angle per position step shrinks with interpolation. In standard RoPE, moving one position rotates dimension pair 0 by 1 radian. With 4x interpolation, the same step rotates by only 0.25 radians. This compression is the trade-off at the heart of Position Interpolation: we maintain familiar absolute angles but reduce the angular resolution between nearby positions.
The reduction in relative angle is uniform across all dimension pairs. Every frequency is scaled by the same factor . This uniformity is both Position Interpolation's strength and its weakness. It is a strength because it is simple, easy to implement, and easy to analyze. It is a weakness because different frequencies have different sensitivities to this scaling. The high-frequency dimensions, which originally distinguished positions that are just a few tokens apart, now must make do with one-quarter of their original angular resolution. The low-frequency dimensions, which already had coarse resolution, are relatively unaffected because they were not being used for fine-grained discrimination in the first place.
