ALiBi › The Core Idea: Penalizing Distance

ALiBi's insight is elegant in its simplicity: do not encode position in the embeddings at all. Instead, modify the attention mechanism itself to prefer nearby tokens over distant ones. Embeddings carry semantic content; let attention carry positional structure.

Think of it as a gravitational field. Every token exerts an attentional "pull" on every other token based on content compatibility. ALiBi adds a distance decay to this pull: the further two tokens are from each other, the weaker the pull, regardless of content. Nearby tokens always have a structural advantage. This is not unlike how we read: we naturally weight recent context more heavily than distant context when interpreting the meaning of a word.

Consider what attention scores represent. They measure compatibility between a query and a key. After softmax, these scores determine how much each position contributes to the output. ALiBi introduces a simple bias that subtracts from the score based on distance. To compute the modified attention score between position $i$ (as the query) and position $j$ (as the key), we start with the standard dot product and then impose the distance penalty:

where:

• $\text{score}_{ij}$: the modified attention score between positions $i$ and $j$
• $\mathbf{q}_i \cdot \mathbf{k}_j$: the original dot product between query $i$ and key $j$, capturing content-based compatibility
• $m$: a slope parameter that controls how aggressively to penalize distance (different for each attention head)
• $|i - j|$: the absolute distance between the two positions in the sequence

The subtraction means distant tokens receive lower scores. A token 10 positions away gets penalized by $10m$, while an adjacent token gets penalized by just $m$. After softmax normalization, this translates to nearby tokens receiving higher attention weights. The key insight is that the penalty is linear in distance: every additional step away costs exactly $m$ more in score, regardless of whether you are stepping from position 1 to position 2 or from position 1000 to position 1001.

This is position encoding without position embeddings. The model learns nothing about absolute position during pretraining of its embeddings. Position information enters only through the attention bias, and only at the moment of computing attention scores. The embeddings remain clean semantic representations of token content, and the relative positional structure is layered on top at attention time.

Why does the penalty have to be linear? The authors experimented with other functional forms, including concave penalties that taper off quickly and convex penalties that grow steeply. Linear penalties produced the best combination of in-distribution performance and out-of-distribution (long-sequence) extrapolation. The linear form also has a pleasing mathematical property: it represents a constant cost per unit of distance, which creates a uniform "discount rate" on attention over distance. This discount rate is consistent regardless of absolute position, which is the property that makes extrapolation work.