Position Interpolation › Implementing Position Interpolation
Language AI HandbookAll chapters Position Interpolation › Implementing Position InterpolationOne section · Sections in this chapter5 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
With the mathematical foundation in place, let's translate Position Interpolation into code. The implementation is remarkably simple: we compute the scale factor, multiply positions by that factor, and then proceed with standard RoPE angle computation.
Computing Interpolated Angles
The core function takes positions along with both the training and target lengths. It computes the scale factor internally, scales all positions, and returns angles that stay within the training range.
def compute_interpolated_angles(
positions, d_model=64, base=10000, train_length=2048, target_length=8192
):
"""Compute position-interpolated RoPE angles.
Args:
positions: Array of position indices
d_model: Embedding dimension
base: Base for frequency computation
train_length: Maximum position seen during training
target_length: Target extended context length
Returns:
angles: Array of shape (len(positions), d_model/2)
"""
# Compute scale factor
scale = train_length / target_length
# Scale positions
positions = np.array(positions)
scaled_positions = positions * scale
# Compute frequencies (unchanged from original RoPE)
dim_pairs = d_model // 2
i = np.arange(dim_pairs)
theta = 1.0 / (base ** (2 * i / d_model))
# Compute angles with scaled positions
angles = np.outer(scaled_positions, theta)
return angles, scale
# Compute interpolated angles for the extended range
interp_angles, scale = compute_interpolated_angles(
extended_positions, train_length=train_length, target_length=extended_length
)Verifying the Angle Bounds
The critical test: do the interpolated angles at the maximum extended position match the training maximum? Let's compare the angles at position 8191 across three methods.
Scale factor: 0.2500 Maximum angles comparison at position 8191: ------------------------------------------------------------ Method Pair 0 Pair 15 Pair 31 ------------------------------------------------------------ Original RoPE 8191.00 109.2287 1.092287 Position Interpolation 2047.75 27.3072 0.273072 Training max (pos 2047) 2047.00 27.2972 0.272972
The results confirm our mathematical derivation. With Position Interpolation, the maximum angles at position 8191 match the training maximum at position 2047 across all dimension pairs. Original RoPE would produce angles 4x larger at the extended position, but Position Interpolation compresses them back into the familiar range.
This verification step is important because it rules out subtle numerical errors. The interpolated angles should match the training maximum within floating-point precision. If they don't, it indicates a bug in the scale factor computation or the position scaling logic.
Let's visualize this comparison as a heatmap to see the pattern across all dimension pairs simultaneously.
The heatmap makes the difference visually striking. The top row shows the intense "heat" of original RoPE's extrapolated angles, especially in the fast-rotating dimensions on the left. Position Interpolation (middle row) shows the same pattern as the training maximum (bottom row), confirming that we've successfully mapped extended positions back into familiar territory.
Visualizing the Position Mapping
Let's visualize this compression graphically. The plot below shows how actual positions in the extended sequence map to effective positions for RoPE computation.
