Position Interpolation › A Worked Example

Before writing code, let's work through a concrete numerical example to solidify the concept. This will make the subsequent implementation much easier to follow.

Suppose we have a model with $d_{\text{model}} = 4$ (two dimension pairs, which is unrealistically small but easy to trace), $\text{base} = 10000$, trained on sequences up to $L_{\text{train}} = 8$ positions. We now want to process a sequence of $L_{\text{target}} = 16$ positions.

The scale factor is:

$$s = \frac{L_{\text{train}}}{L_{\text{target}}} = \frac{8}{16} = 0.5$$

The base frequencies for our two dimension pairs ($i = 0$ and $i = 1$ with $d = 4$) are:

$$\theta_0 = \frac{1}{10000^{0/4}} = \frac{1}{10000^{0}} = 1.0 \text{ rad/position}$$ $$\theta_1 = \frac{1}{10000^{2/4}} = \frac{1}{100} = 0.01 \text{ rad/position}$$

Now consider position $m = 12$ in the extended sequence. Under standard RoPE (no interpolation), the rotation angles would be:

• Dimension pair 0: $12 \times 1.0 = 12.0$ radians
• Dimension pair 1: $12 \times 0.01 = 0.12$ radians

The training maximum (at position 7) was:

• Dimension pair 0: $7 \times 1.0 = 7.0$ radians
• Dimension pair 1: $7 \times 0.01 = 0.07$ radians

Position 12 exceeds the training maximum for both dimension pairs. The fast-rotating dimension pair 0 is especially far outside the training range (12.0 vs 7.0 radians).

Under Position Interpolation, we scale the position first: $m' = 12 \times 0.5 = 6.0$. The rotation angles become:

• Dimension pair 0: $6.0 \times 1.0 = 6.0$ radians (within training max of 7.0)
• Dimension pair 1: $6.0 \times 0.01 = 0.06$ radians (within training max of 0.07)

Both angles now fall within the range the model saw during training. The model can apply its learned interpretation of these rotation patterns to process what is effectively a mid-sequence position. The trade-off is that position 12 in the extended sequence "feels like" position 6 in the original sequence, meaning the model perceives it as occupying roughly the middle of the context rather than three-quarters of the way through. Fine-tuning corrects this calibration.