ALiBi › Historical Context

ALiBi was introduced in the paper "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" by Ofir Press, Noah A. Smith, and Mike Lewis, published in 2022 at ICLR. The title captures the core contribution succinctly: train on short sequences, test (inference) on long sequences, enabled by the linear bias design.

The paper was motivated by the observation that all existing position encoding methods (sinusoidal, learned, T5 relative biases, and RoPE) failed to generalize beyond training sequence length. The authors systematically evaluated length extrapolation and found that ALiBi dramatically outperformed all competitors on this metric while remaining competitive on standard perplexity benchmarks.

ALiBi was adopted by the BLOOM project (the largest open multilingual language model at the time of its release in 2022 with 176 billion parameters) and by MosaicML's MPT family of models. This adoption showed that ALiBi could support production-scale language models. The method influenced subsequent thinking about position encoding design and contributed to the recognition that inductive biases favoring locality are valuable, not limiting.