Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

ALiBi › Head-Specific Slopes

Michael BrenndoerferUpdated May 30, 20255 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 101ALiBi › Head-Specific Slopes

Reading section 5 of 16 · Head-Specific Slopes · 5 minSection 5 of 16 · 5 min

Section 5 of 16 · 5 min read

ALiBi › Head-Specific SlopesOne section · Sections in this chapter5 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A single slope would force all attention heads to have the same locality preference. That would be limiting: different aspects of language operate at different scales, and a model benefits from heads that specialize in different scopes. Some heads might focus on very local context (steep slope, harsh penalties for distance), while others maintain broader receptive fields (gentle slope, mild penalties).

Think of it like a camera system with multiple lenses. A telephoto lens captures fine detail up close but misses the broader scene. A wide-angle lens captures the full picture but loses local detail. You want both working together. ALiBi gives each attention head a different "focal length" by assigning it a different slope. The ensemble of heads covers a wide spectrum of temporal scales, from immediate local syntax to long-range topic coherence.

The slopes are not learned but fixed according to a geometric sequence. The decision to fix them (rather than learn them) is deliberate: it eliminates hyperparameters from the position encoding, makes the encoding fully deterministic and reproducible, and avoids the risk of the model learning pathological slope values during training. For a model with $h$ attention heads, the slope for head $i$ is:

m_i = \frac{1}{2^{\frac{8}{h} \cdot i}} \quad \text{for } i = 1, 2, \ldots, h

where:

$m_i$ : the slope parameter for attention head $i$
$h$ : the total number of attention heads in the model
$i$ : the head index, ranging from 1 to $h$
$\frac{8}{h}$ : a scaling factor that ensures slopes span a consistent range regardless of how many heads the model has
$2^{\frac{8}{h} \cdot i}$ : the denominator that grows exponentially with head index, making later heads have gentler slopes

Why does this formula make sense? Notice that the exponent $\frac{8}{h} \cdot i$ ranges from $\frac{8}{h}$ (head 1) to $8$ (head $h$ ). The denominator is therefore always $2^{\frac{8}{h}}$ through $2^8 = 256$ , independent of $h$ . This ensures that the range of slopes remains consistent across model architectures with different head counts. Whether you have 4 heads or 32 heads, the steepest head always has a slope near $1/2^{8/h}$ and the gentlest head always has a slope near $1/256 \approx 0.004$ .

The base of 8 in the exponent was chosen empirically by the ALiBi authors. With 8 heads, the exponent simplifies to just $i$ , giving slopes of $\frac{1}{2^1}, \frac{1}{2^2}, \ldots, \frac{1}{2^8}$ , which equals $0.5, 0.25, 0.125, \ldots, 0.00390625$ . With other head counts, the formula interpolates appropriately.

In[8]:

Code

def get_alibi_slopes(num_heads):
    """
    Compute ALiBi slopes for each attention head.

    The slopes follow a geometric sequence, with the first head
    having the steepest slope (most local attention) and the last
    head having the gentlest slope (broadest attention).

    Args:
        num_heads: Number of attention heads

    Returns:
        Array of slopes, one per head
    """
    # Compute the ratio for the geometric sequence
    ratio = 2 ** (8 / num_heads)

    # Generate slopes: 1/ratio, 1/ratio^2, ..., 1/ratio^num_heads
    slopes = 1.0 / (ratio ** np.arange(1, num_heads + 1))

    return slopes

Out[9]:

Console

4 heads: slopes = [0.25     0.0625   0.015625 0.003906]
8 heads: slopes = [0.5      0.25     0.125    0.0625   0.03125  0.015625 0.007812 0.003906]
16 heads: slopes = [0.707107 0.5      0.353553 0.25     0.176777 0.125    0.088388 0.0625
 0.044194 0.03125  0.022097 0.015625 0.011049 0.007812 0.005524 0.003906]

The geometric progression ensures that slopes span several orders of magnitude. With 8 heads, the steepest slope (0.5) penalizes a distance of 10 by 5 logits, effectively eliminating distant tokens from consideration after softmax. The gentlest slope (roughly 0.004) penalizes the same distance by only 0.04 logits, allowing the head to attend broadly across hundreds of positions. This spread is what gives ALiBi its multi-scale character.

Notice that the formula produces a different set of slopes for 4, 8, and 16 heads, but in each case, the slopes span roughly the same overall range from steep to gentle. With 4 heads, you get fewer granular distinctions but the extremes remain similar. With 16 heads, you get finer granularity, more heads at intermediate slopes, but still roughly the same endpoints. This consistency is the purpose of the $8/h$ scaling factor.

Out[10]:

Visualization

Bar chart showing ALiBi slopes for 8 attention heads, with values decreasing geometrically from 0.5 for head 1 to near zero for head 8. — ALiBi slopes across 8 attention heads, shown on a linear scale. The geometric progression creates a rapid decay from head 1 (slope 0.5, strong locality bias) to head 8 (slope near 0.004, broad attention). Each head effectively operates on a different spatial scale, allowing the model to capture both local and distant dependencies simultaneously.