Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

ALiBi › Visualizing the Attention Bias

Michael BrenndoerferUpdated May 30, 20253 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 101ALiBi › Visualizing the Attention Bias

Reading section 6 of 16 · Visualizing the Attention Bias · 3 minSection 6 of 16 · 3 min

Section 6 of 16 · 3 min read

ALiBi › Visualizing the Attention BiasOne section · Sections in this chapter6 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The slope values are informative in the abstract, but seeing what they produce as attention bias matrices makes the effect concrete. Let's visualize how ALiBi biases shape attention patterns for the steepest and gentlest heads:

Out[11]:

Visualization

Heatmap showing ALiBi bias for head 1 with steep slope, displaying strong negative values far from the diagonal. — ALiBi bias matrix for head 1 (slope 0.5, steepest). Strong negative values appear just a few positions from the diagonal, creating a narrow attention window. A query at position 18 attending to position 8 (ten steps away) faces a bias of -5.0, effectively eliminating that token from consideration after softmax.

Out[12]:

Visualization

Heatmap showing ALiBi bias for head 8 with gentle slope, displaying mild penalties across all distances. — ALiBi bias matrix for head 8 (slope 0.0039, gentlest). The penalties are so mild that even tokens 19 positions away face a bias of only -0.074. This head can attend broadly across the entire available context with minimal positional discounting.

The contrast is stark and instructive. Head 1's bias matrix shows deep red (strongly negative) values just a few positions from the diagonal. By position 10, the penalty exceeds -5 logits, making those tokens nearly invisible after softmax unless their content score is extraordinarily strong. The effective attention window for this head is perhaps 3 to 5 tokens wide. Head 8, in contrast, shows nearly uniform mild penalties throughout. Even at distance 19, the penalty is less than 0.1 logits, allowing meaningful attention to distant tokens across essentially the entire sequence.

This division of labor is intentional and reflects something real about natural language. Linguistic phenomena operate at different scales. Adjacent tokens matter for subject-verb agreement, determiner-noun matching, and immediate syntactic structure. Tokens a few steps away matter for multi-word expressions, phrasal dependencies, and clause-level coherence. Tokens many steps away matter for coreference resolution (a pronoun referring back to a noun introduced many sentences earlier), topic consistency, and document-level discourse structure. By giving different heads different locality preferences, ALiBi enables the model to capture phenomena at multiple scales simultaneously without requiring the model to learn these scales from scratch.