ALiBi › Key Parameters

When implementing ALiBi in your own models, the following parameters control its behavior:

• num_heads: The number of attention heads in your model. ALiBi automatically computes slopes for each head using the geometric sequence formula. More heads create finer granularity in locality preferences: with 16 heads you get smoother coverage of the slope range than with 4 heads.

• slope (m): The penalty strength for each head. Steeper slopes (larger values like 0.5) create strong locality bias where attention concentrates on nearby tokens. Gentler slopes (smaller values like 0.004) allow broader attention across the sequence. These are fixed by the formula rather than tuned, which eliminates a hyperparameter while potentially leaving some performance on the table for specialized tasks.

• causal: Whether to apply causal masking. Set to True for decoder-style autoregressive models where tokens can only attend to previous positions. Set to False for encoder-style bidirectional attention, where both preceding and following tokens are available.

• Base value (8): The constant in the slope formula $m_i = 1/2^{(8/h) \cdot i}$ that controls the range of slopes. The original ALiBi paper uses 8, which ensures slopes span several orders of magnitude regardless of head count. Increasing this base would compress all slopes toward smaller values (gentler biases), while decreasing it would shift toward steeper slopes. The value 8 was selected empirically and is almost never modified in practice.