ALiBi › The Extrapolation Problem

Before understanding ALiBi's solution, we need to appreciate the problem it solves. Transformers trained on sequences of length $L$ often fail dramatically when given sequences of length $2L$ or beyond. This is not a subtle degradation: performance can collapse entirely, with the model producing incoherent outputs for inputs only slightly longer than the training window.

With sinusoidal encodings, the encoding at position 1025 is a specific vector of cosines and sines that the model was never trained to use. The model's linear and attention layers have developed weights that respond appropriately to positions 0 through 1024, but position 1025 looks foreign, and the model has no basis for interpreting it correctly. With learned position embeddings, the situation is even worse: positions beyond the training length simply do not exist in the embedding table. The model cannot process them at all without modification.

RoPE handles this more gracefully because its rotations are defined by a mathematical formula rather than learned weights. In theory, you can compute a rotation for any position. In practice, however, the model's layers have been tuned on rotation patterns corresponding to positions 0 through $L$, and rotation angles at positions far beyond $L$ can fall into regions of the complex plane that the model has never been trained to interpret. Research teams working with RoPE models often need additional techniques like position interpolation or NTK-aware scaling to extend the training context length.

Why does extrapolation matter so much in practice? First, training on very long sequences is extraordinarily expensive. Attention's quadratic complexity means that doubling the sequence length quadruples the attention computation. If you could train on sequences of 1,024 tokens and reliably deploy on sequences of 4,096 or 8,192 tokens, you would save enormous training costs. Second, real-world inputs are unpredictable. A conversational AI might encounter a long document, a code model might see a large file, and a summarization system might receive a lengthy article. A model that degrades gracefully on longer inputs is far more practically valuable than one that requires the input to stay within a hard length ceiling.

The extrapolation problem is essentially a generalization problem across a dimension that most machine learning practitioners don't spend much time thinking about: the sequence length dimension. We routinely generalize across input content, but generalizing across input structure, specifically the sequential length structure, requires position encodings that work by relative relationship rather than absolute coordinate.