ALiBi › ALiBi: Attention with Linear Biases

In the previous chapters, we explored various approaches to encoding position information: sinusoidal encodings that add fixed patterns to embeddings, learned position embeddings that train position vectors from scratch, relative position encodings that capture pairwise distances, and RoPE that rotates embeddings based on position. Each method has trade-offs involving complexity, performance, and the ability to handle sequences longer than those seen during training. We have seen that some methods are elegant but fragile at long range, and others are flexible but costly to implement or interpret.

ALiBi (Attention with Linear Biases) takes a radically different approach. Instead of modifying embeddings or inventing clever rotation schemes, ALiBi simply subtracts a value from attention scores based on the distance between tokens. The farther apart two tokens are, the larger the penalty. This remarkably simple idea, introduced by Press and Smith along with Lewis in 2022, achieves strong performance while enabling something the other methods struggle with: extrapolation to sequences far longer than anything seen during training.

Think of ALiBi as adding a proximity premium to attention. Every pair of tokens starts with a content-based score. This reflects how relevant their representations are to each other. ALiBi then applies a distance tax: nearby tokens pay very little, while distant tokens pay heavily. The model never needs to learn what position 512 means, because position is never represented as a vector. Instead, the model learns to work with attention patterns where nearby context is systematically favored. That behavioral pattern stays consistent no matter how long the sequence grows.

The key insight is that position information can enter the model through the mechanism of attention itself, rather than through the token representations. Every prior method we examined grafted position onto the input before it reached the attention layers. ALiBi asks a different question: why not inject position directly where positional relationships matter? Attention is where the model decides which tokens to look at, so shaping attention by distance is a direct and transparent way to express locality.

This design choice has several consequences. First, the model's token embeddings remain pure semantic representations, uncontaminated by position. Second, the position bias is fully interpretable: you can look at any attention head and immediately understand its locality preference by inspecting its slope. Third, and most importantly, the bias depends only on relative distance, which means the model has never seen a "position 1024" encoding and so does not need one during inference at length 2048.

In this chapter, we will work through ALiBi from the ground up. We will develop the mathematical formulation step by step, implement it in code, visualize how different heads behave, and understand exactly why extrapolation works. We will also compare ALiBi to RoPE, discuss practical trade-offs, and examine which real-world models have adopted it.