ALiBi › Practical Considerations

When deploying ALiBi in your own models, several practical considerations affect performance and behavior. Understanding these factors helps you make informed decisions about whether ALiBi is the right choice and how to configure it.

ALiBi's computational overhead relative to standard attention is minimal. The bias matrix computation is $O(n^2)$, matching the overall cost of attention, and the actual operation (a matrix add) is extremely fast on modern hardware. There are no additional parameters to initialize, no embedding lookups, and no trigonometric operations. In a typical transformer implementation, adding ALiBi increases the attention computation time by perhaps 1 to 2 percent, well within noise.

The fixed slopes, while convenient, raise a natural question: should they be learned instead? Several research papers have explored learned ALiBi slopes. The findings are mixed: on standard benchmarks within the training length, learned slopes sometimes improve performance slightly. But on out-of-distribution sequence lengths (the key use case for ALiBi), learned slopes may not extrapolate as reliably as the fixed formula. The fixed slopes guarantee consistent behavior across all sequence lengths by design. Learned slopes might overfit to the training length range. If you decide to experiment with learned slopes, treat them as a carefully monitored hyperparameter rather than a drop-in replacement.

The causal versus bidirectional distinction matters for deployment. Decoder-only language models (for text generation) use causal ALiBi, where the upper triangle of the bias matrix is $-\infty$. Encoder models (for classification, understanding) use bidirectional ALiBi, where the full symmetric bias matrix applies and both left and right context receive distance penalties. Encoder-decoder models (for translation, summarization) typically use bidirectional ALiBi in the encoder and causal ALiBi in the decoder. The mathematics remains the same in all cases; only which entries of the bias matrix are live changes.

The effective context window visualization reveals a striking difference across heads. Head 1 drops below the 10 percent relative weight threshold very quickly, around distance 5, because its steep slope ($m = 0.5$) imposes an exponentially decaying effective weight. Head 4 with slope 0.0625 reaches the 10 percent threshold around distance 37. Head 8 with slope 0.0039 barely crosses it even at distance 100. When all heads work together, the model simultaneously has access to very local structure (from head 1), medium-range context (from heads 2 through 7), and long-range dependencies (from head 8). This is arguably more structured than what emerges organically in models without explicit position encoding.