ALiBi › Why ALiBi Extrapolates

The key to ALiBi's extrapolation ability lies in what the model learns during training. With other position encoding schemes, the model learns to interpret specific position representations. A sinusoidal encoding for position 512 produces a particular vector that the model has seen and learned to use. A learned position embedding for position 512 is a specific trained vector. When you encounter position 1025, this is a novel representation the model has not seen, and its layers have no well-defined response to it.

ALiBi sidesteps this problem entirely. The model never learns position representations because there are none to learn. Instead, it learns to work with relative attention patterns shaped by the linear bias. During training on sequences of length 1024, the model sees attention patterns where nearby tokens are favored and distant tokens are penalized. This is true at position 10, position 500, and position 1000. At every position in the training range, the model observes the same structural pattern: recent tokens have higher weight, distant tokens have lower weight.

When inference extends to position 2048, the same principle applies. The local neighborhood still receives favorable bias. Tokens 10 positions away still get penalized by the same amount. The absolute positions are larger, but the relative structure is completely unchanged. The model has learned to extract information from attention patterns that favor locality, and those patterns remain consistent regardless of sequence length.

There is a subtle but important point here. When a model processes a long sequence at inference time, tokens near the beginning of the sequence will be far from the current query position. ALiBi will penalize them heavily, potentially so heavily that they receive near-zero attention weight. This is not a failure: it is the expected behavior. The model is effectively operating with a soft window: it pays close attention to recent context and discounts the distant past. For sequences significantly longer than the training length, the very first tokens may be practically invisible to later queries. Whether this hurts performance depends on the task: for many language modeling and generation tasks, the most relevant context is recent, so this graceful distance falloff can improve performance.

Let's visualize the consistency property that makes extrapolation work:

The curves overlap perfectly because ALiBi's bias depends only on distance, never on absolute position. Whether the query is at token 20 or token 5000, the bias for a key 30 positions back is the same: $-30m$. This is the mathematical basis for length extrapolation, and it is also the mathematical reason why it could not work for sinusoidal or learned position embeddings: those methods embed absolute position, so the representation at position 5000 is fundamentally different from (and trained less on than) the representation at position 20.