ALiBi: Attention with Linear Biases for Position Encoding

In the previous chapters, we explored various approaches to encoding position information: sinusoidal encodings that add fixed patterns to embeddings, learned position embeddings that train position vectors from scratch, relative position encodings that capture pairwise distances, and RoPE that rotates embeddings based on position. Each method has trade-offs involving complexity, performance, and the ability to handle sequences longer than those seen during training.

ALiBi (Attention with Linear Biases) takes a radically different approach. Instead of modifying embeddings or inventing clever rotation schemes, ALiBi simply subtracts a value from attention scores based on the distance between tokens. The farther apart two tokens are, the larger the penalty. This remarkably simple idea, introduced by Press et al. in 2022, achieves strong performance while enabling something the other methods struggle with: extrapolation to sequences far longer than anything seen during training.

Before understanding ALiBi's solution, we need to appreciate the problem it solves. Transformers trained on sequences of length $L$ often fail dramatically when given sequences of length $2L$ or beyond. The sinusoidal encodings produce positions the model has never seen. Learned position embeddings simply don't exist for positions beyond the training range. Even RoPE, despite its theoretical elegance, can struggle with extreme extrapolation.

Why does this matter? Training on very long sequences is computationally expensive due to attention's quadratic complexity. If we could train on shorter sequences and deploy on longer ones, we'd save enormous resources. More practically, real-world applications often encounter documents, conversations, or code files that exceed training lengths. A model that degrades gracefully on longer inputs is far more useful than one that fails catastrophically.

ALiBi's insight is elegant in its simplicity: don't encode position in the embeddings at all. Instead, modify the attention mechanism itself to prefer nearby tokens over distant ones.

Consider what attention scores represent: they measure compatibility between a query and a key. After softmax, these scores determine how much each position contributes to the output. ALiBi introduces a simple bias that subtracts from the score based on distance:

where:

• $\text{score}_{ij}$: the modified attention score between positions $i$ and $j$
• $\mathbf{q}_i \cdot \mathbf{k}_j$: the original dot product between query $i$ and key $j$
• $m$: a slope parameter that controls how aggressively to penalize distance
• $|i - j|$: the absolute distance between the two positions

The subtraction means distant tokens receive lower scores. A token 10 positions away gets penalized by $10m$, while an adjacent token gets penalized by just $m$. After softmax normalization, this translates to nearby tokens receiving higher attention weights.

This is position encoding without position embeddings. The model learns nothing about position during pretraining of the embeddings themselves. Position information enters only through the attention bias, and only at the moment of computing attention scores.

Now that we understand ALiBi's core insight, let's develop the complete mathematical framework. We'll build from standard attention to ALiBi-augmented attention, showing exactly where and how the position bias enters the computation.

Recall that standard self-attention operates on three matrices derived from the input sequence. For a sequence of $n$ tokens:

where:

• $Q \in \mathbb{R}^{n \times d_k}$: the query matrix containing query vectors for all $n$ positions
• $K \in \mathbb{R}^{n \times d_k}$: the key matrix containing key vectors for all positions
• $V \in \mathbb{R}^{n \times d_v}$: the value matrix containing value vectors for all positions
• $d_k$: the dimension of queries and keys
• $QK^T \in \mathbb{R}^{n \times n}$: the raw attention score matrix where entry $(i, j)$ is the dot product between query $i$ and key $j$
• $\sqrt{d_k}$: scaling factor that prevents dot products from growing too large in high dimensions

The matrix $QK^T$ captures content-based similarity: how much each query "wants" to attend to each key based purely on their learned representations. But this matrix is blind to position. Token 1 attending to token 2 produces the same score whether they're adjacent or separated by 500 tokens. This is where ALiBi intervenes.

ALiBi's modification is surgical. Rather than changing how queries, keys, or values are computed, it adds a single term to the attention scores:

The bias matrix $B \in \mathbb{R}^{n \times n}$ is added directly to the scaled scores before softmax. This is the only change to standard attention. The matrix $B$ encodes position information through a remarkably simple rule:

where:

• $B_{ij}$: the bias added to the attention score between query position $i$ and key position $j$
• $m$: the slope parameter for this attention head (controls penalty strength)
• $|i - j|$: the absolute distance between positions $i$ and $j$

Let's unpack why this formula works. The absolute distance $|i - j|$ measures how far apart two positions are in the sequence. Adjacent tokens have distance 1, tokens separated by 10 positions have distance 10. Multiplying by a positive slope $m$ and negating creates a penalty that grows linearly with distance.

Consider what happens for a query at position 5 attending to various keys:

• Attending to position 5 (itself): $B_{55} = -m \cdot 0 = 0$ (no penalty)
• Attending to position 4 (adjacent): $B_{54} = -m \cdot 1 = -m$ (small penalty)
• Attending to position 0 (distant): $B_{50} = -m \cdot 5 = -5m$ (large penalty)

The negative bias reduces the attention score, and larger distances produce more negative biases. After softmax normalization, this translates to lower attention weights for distant tokens. The content-based similarity in $QK^T$ still matters, but now it competes against a distance penalty.

For decoder-style models that generate text left-to-right, we need causal masking: position $i$ can only attend to positions $j \le i$. The combined bias matrix looks like:

The structure reveals two distinct components:

1. Lower triangle (including diagonal): Contains the ALiBi distance penalties. The diagonal is 0 (self-attention has no penalty), and values increase as you move left (farther from the query position).

2. Upper triangle: Contains $-\infty$ values from the causal mask. These become 0 after softmax, preventing any attention to future positions.

The $-\infty$ entries ensure that after $\exp(-\infty) = 0$, future positions contribute nothing to the weighted sum. The linear penalties in the lower triangle shape how attention flows among the allowed past positions.

Let's translate this mathematics into code, building up the bias matrix piece by piece.

The key insight is using NumPy broadcasting to compute all pairwise distances at once. By reshaping the position array into a column vector positions[:, None] and a row vector positions[None, :], subtraction produces an $n \times n$ matrix where entry $(i, j)$ is $i - j$. Taking the absolute value gives us the distance matrix.

Let's see what this produces for a small sequence:

Reading this matrix: the diagonal is zero because a token attending to itself has distance zero. Moving away from the diagonal in either direction, penalties grow linearly. Position 5 (row 5) attending to position 0 (column 0) shows $-2.5$, exactly $-0.5 \times 5$ as expected. The matrix is symmetric because distance is symmetric: $|i - j| = |j - i|$.

For decoder models, we overlay the causal mask:

The np.triu function creates an upper triangular matrix of ones, which we use to identify positions that should be masked. The k=1 argument excludes the diagonal, since a token should be able to attend to itself.

Now the upper triangle shows inf (NumPy's display for $-\infty$), while the lower triangle retains the distance penalties. Row 5 can attend to all previous positions with penalties $-2.5, -2.0, -1.5, -1.0, -0.5, 0.0$ for positions 0 through 5 respectively. Row 0 can only attend to itself with penalty 0.

Let's visualize both the distance matrix and the resulting causal ALiBi bias matrix side by side:

The left matrix shows raw distances: the diagonal is 0 (same position), adjacent cells are 1, and values grow as positions diverge. The right matrix shows what happens after applying ALiBi: distances become negative penalties (scaled by the slope), and the upper triangle is masked to enforce causality.

ALiBi uses different slope values for different attention heads. This is crucial: a single slope would force all heads to have the same locality preference, but different heads benefit from different perspectives. Some heads might focus on very local context (steep slope, harsh penalties for distance), while others maintain broader receptive fields (gentle slope, mild penalties).

The slopes are not learned but fixed according to a geometric sequence. For a model with $h$ attention heads, the slope for head $i$ is:

where:

• $m_i$: the slope parameter for attention head $i$
• $h$: the total number of attention heads in the model
• $i$: the head index, ranging from 1 to $h$
• $\frac{8}{h}$: a scaling factor that ensures slopes span a consistent range regardless of head count
• $2^{\frac{8}{h} \cdot i}$: the denominator that grows exponentially with head index

The base of 8 in the exponent was chosen empirically by the ALiBi authors. It ensures that slopes span roughly 8 orders of magnitude when divided across the heads. With 8 heads, the exponent simplifies to just $i$, giving slopes of $\frac{1}{2^1}, \frac{1}{2^2}, \ldots, \frac{1}{2^8}$, which equals $0.5, 0.25, 0.125, \ldots, 0.00390625$.

The geometric progression ensures that slopes span several orders of magnitude. With 8 heads, the steepest slope (0.5) penalizes a distance of 10 by 5 logits, effectively eliminating distant tokens from consideration. The gentlest slope (roughly 0.004) penalizes the same distance by only 0.04 logits, allowing the head to attend broadly.

Let's visualize how ALiBi biases affect attention patterns for different heads:

The contrast is stark. Head 1's bias matrix shows deep blue (strongly negative) values just a few positions from the diagonal. By position 10, the penalty exceeds -5 logits, making those tokens nearly invisible after softmax. Head 8, in contrast, shows mild penalties throughout. Even at distance 19, the penalty is less than 0.1 logits, allowing meaningful attention to distant tokens.

This division of labor is intentional. Linguistic phenomena operate at different scales: adjacent tokens matter for syntax and local coherence, while distant tokens matter for coreference, topic consistency, and long-range dependencies. By giving different heads different locality preferences, ALiBi enables the model to capture phenomena at multiple scales simultaneously.

Now let's implement ALiBi-augmented attention from scratch:

Let's test this implementation with a simple example:

Compare the attention patterns between heads. Head 1 with its steep slope concentrates attention heavily on recent positions, with weights dropping rapidly as distance increases. Head 4 with its gentle slope distributes attention more evenly across the available context.

To understand the impact of ALiBi more directly, let's compare attention patterns with and without the position bias. We'll compute attention for the same queries and keys, once with standard attention (no position encoding) and once with ALiBi:

The difference is striking. Standard attention distributes weight based purely on content similarity, sometimes attending strongly to distant positions. ALiBi reshapes this pattern, pulling attention toward recent tokens while still allowing content to influence the final distribution. This locality bias emerges from a single matrix addition, requiring no learned position embeddings.

The key to ALiBi's extrapolation ability lies in what the model learns during training. With other position encoding schemes, the model learns to interpret specific position representations. A sinusoidal encoding for position 512 produces a particular pattern that the model has seen and learned to use. A learned position embedding for position 512 is a specific trained vector. When you encounter position 513 or 1024, these are novel representations the model hasn't seen.

ALiBi sidesteps this problem entirely. The model never learns position representations because there are none. Instead, it learns to work with relative attention patterns shaped by the linear bias. During training on sequences of length 1024, the model sees attention patterns where nearby tokens are favored and distant tokens are penalized. This is true at position 10, position 500, and position 1000.

When inference extends to position 2048, the same principle applies. The local neighborhood still receives favorable bias. Tokens 10 positions away still get penalized by the same amount. The absolute positions are larger, but the relative structure is unchanged. The model has learned to extract information from attention patterns that favor locality, and those patterns remain consistent regardless of sequence length.

Let's visualize this consistency:

The curves overlap perfectly because ALiBi's bias depends only on distance, never on absolute position. This is the mathematical basis for length extrapolation.

Both ALiBi and RoPE are widely used in modern language models, and both handle relative position. But they take fundamentally different approaches:

RoPE encodes position by rotating query and key vectors. When computing dot products, the rotation angles combine such that the result depends on relative position. This is elegant and theoretically motivated, but the rotation patterns at unseen positions can behave unexpectedly.

ALiBi's approach is more brutalist: just penalize distance. There's no rotation, no complex number interpretation, no interplay between embedding dimensions. The simplicity has practical benefits. ALiBi requires no changes to the embedding pipeline and adds minimal computational overhead.

Let's compare implementation complexity:

The ALiBi bias is a single line of code added to standard attention. RoPE requires restructuring how queries and keys are computed. Both work, but ALiBi's simplicity is a genuine advantage for implementation and debugging.

When should you choose ALiBi? Consider these factors:

Training efficiency. ALiBi adds minimal overhead. The bias computation is a simple matrix operation that's negligible compared to the attention computation itself. There are no additional parameters to train.

Length flexibility. If your deployment might encounter sequences longer than training, ALiBi provides a safety net. Models like BLOOM and MPT use ALiBi partly for this reason.

Simplicity. ALiBi is easy to implement correctly. The fixed slopes mean no hyperparameter tuning for the position encoding itself. Debugging attention patterns is straightforward because the bias is directly interpretable.

Limitations. ALiBi's linear penalty may not capture all positional relationships. Some tasks might benefit from RoPE's more nuanced encoding. Extremely local heads (with steep slopes) can struggle if a task genuinely requires attending to distant tokens with equal strength.

In practice, many production models use either ALiBi or RoPE, with the choice often depending on the team's preferences and empirical results on target tasks. Both represent significant advances over sinusoidal encodings and learned position embeddings.

ALiBi's simplicity is both its strength and its limitation. The linear penalty assumes that relevance decreases monotonically with distance, which is generally true for language but not universally. Some linguistic phenomena, like matching parentheses in code or tracking long-distance agreement in legal documents, might benefit from attention patterns that don't decay linearly.

The fixed slopes, while eliminating hyperparameters, also remove flexibility. A model cannot learn task-specific locality preferences through the position encoding itself. If a particular downstream task requires unusual attention patterns, the model must learn to overcome the ALiBi bias through the content-based attention scores.

Despite these limitations, ALiBi has proven remarkably effective. The BLOOM family of models, including the 176-billion parameter BLOOM-176B, uses ALiBi. So does MPT (MosaicML Pretrained Transformer). These models demonstrate that ALiBi scales to the largest models and handles diverse tasks well.

The impact of ALiBi extends beyond its direct use. It demonstrated that position encoding can be far simpler than previously thought. The original Transformer's sinusoidal encodings were ingenious but perhaps overengineered for the task. ALiBi showed that a linear penalty on distance, applied at attention time, is sufficient for strong performance. This insight has influenced subsequent work on efficient transformers and position encoding schemes.

When implementing ALiBi in your own models, these parameters determine behavior:

• num_heads: The number of attention heads in your model. ALiBi automatically computes slopes for each head using the geometric sequence formula. More heads create finer granularity in locality preferences.

• slope (m): The penalty strength for each head. Steeper slopes (larger values like 0.5) create strong locality bias where attention concentrates on nearby tokens. Gentler slopes (smaller values like 0.004) allow broader attention across the sequence. These are fixed by the formula, not tuned.

• causal: Whether to apply causal masking. Set to True for decoder-style autoregressive models where tokens can only attend to previous positions. Set to False for encoder-style bidirectional attention.

• Base value (8): The constant in the slope formula $m_i = 1/2^{(8/h) \cdot i}$ that controls the range of slopes. The original ALiBi paper uses 8, which ensures slopes span several orders of magnitude. This value is typically not modified.

ALiBi offers a refreshingly simple approach to position encoding:

• No position embeddings. Position enters only through attention biases, not through modifications to token representations.

• Linear distance penalty. Attention scores are reduced by $m \cdot |i - j|$, where $m$ is a head-specific slope and $|i - j|$ is the distance between query position $i$ and key position $j$. Nearby tokens are favored, distant tokens are penalized.

• Geometric slopes. Different attention heads use different slopes, creating multi-scale attention. Some heads focus locally, others attend broadly.

• Strong extrapolation. Because only relative distance matters (not absolute position), models trained on short sequences can process longer sequences at inference time.

• Minimal overhead. ALiBi adds one matrix addition to attention computation. No additional parameters, no complex rotation arithmetic.

The next chapter will compare all position encoding methods we've covered: sinusoidal, learned, relative, RoPE, and ALiBi. You'll see how each handles key challenges like extrapolation, computational cost, and representational power.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about ALiBi (Attention with Linear Biases).