ALiBi › Summary

ALiBi offers a refreshingly simple approach to position encoding that achieves strong performance and excellent length extrapolation through a single design insight: replace position embeddings with a direct penalty on attention score based on distance.

The key ideas in this chapter are:

• No position embeddings. Position enters only through attention biases, not through modifications to token representations. This keeps embeddings clean and makes position encoding completely separable from content encoding.

• Linear distance penalty. Attention scores are reduced by $m \cdot |i - j|$, where $m$ is a head-specific slope and $|i - j|$ is the distance between query position $i$ and key position $j$. Nearby tokens are favored, distant tokens are penalized, and the penalty is always proportional to distance.

• Geometric slopes. Different attention heads use different slopes, creating multi-scale attention. Some heads focus locally on syntax and immediate context, others attend broadly to long-range dependencies. The slopes are fixed by formula, not learned.

• Strong extrapolation. Because only relative distance matters (not absolute position), models trained on short sequences can process longer sequences at inference time. The bias structure is exactly the same at position 100 as at position 5000.

• Minimal overhead. ALiBi adds one matrix addition to attention computation. There are no additional parameters, no complex rotation arithmetic, and no changes to the embedding pipeline.

The next chapter will compare the position encoding methods we have covered, from sinusoidal and learned encodings to relative approaches such as RoPE and ALiBi. You will see how each handles key challenges like extrapolation, computational cost, and representational power, and when to choose each method for real-world applications.