Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multi-Head Attention › Multi-Head vs Single-Head: An Empirical Comparison

Michael BrenndoerferUpdated May 30, 20253 min read

Language AI HandbookAll chapters

Part 13 of 67Self-Attention

Chapter 94Multi-Head Attention › Multi-Head vs Single-Head: An Empirical Comparison

Reading section 9 of 14 · Multi-Head vs Single-Head: An Empirical Comparison · 3 minSection 9 of 14 · 3 min

Section 9 of 14 · 3 min read

Multi-Head Attention › Multi-Head vs Single-Head: An Empirical ComparisonOne section · Sections in this chapter9 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

How much does multi-head attention help? Let's compare the representational capacity of single-head versus multi-head attention on a simple task: capturing diverse pairwise relationships.

In[17]:

Code

def compute_attention_diversity(weights, num_heads):
    """
    Measure how different the attention patterns are across heads.
    Higher values indicate more diverse (specialized) heads.
    """
    if num_heads == 1:
        return 0.0

    # Compute pairwise Jensen-Shannon divergence between heads
    from scipy.spatial.distance import jensenshannon

    total_div = 0
    count = 0
    for i in range(num_heads):
        for j in range(i + 1, num_heads):
            # Average JS divergence across all query positions
            for q in range(weights.shape[2]):
                div = jensenshannon(weights[0, i, q], weights[0, j, q])
                if not np.isnan(div):
                    total_div += div
                    count += 1

    return total_div / count if count > 0 else 0.0


# Compare 1, 2, 4, 8 heads with the same total dimension
d_model = 64
head_configs = [1, 2, 4, 8]
diversity_scores = []

np.random.seed(42)
x = np.random.randn(1, 8, d_model)

for num_heads in head_configs:
    mha = MultiHeadAttention(d_model, num_heads, seed=42)
    _, weights = mha.forward(x)
    diversity = compute_attention_diversity(weights, num_heads)
    diversity_scores.append(diversity)

Out[18]:

Visualization

Bar chart showing increasing diversity scores from 1 to 8 heads. — Attention pattern diversity increases with the number of heads. With more heads, the model can simultaneously capture more distinct relationship types, though returns diminish as heads begin to overlap in function.

Out[19]:

Console

Diversity scores by number of heads:
  1 head(s): 0.0000
  2 head(s): 0.5962
  4 head(s): 0.5770
  8 head(s): 0.5774

The diversity metric reveals a clear pattern: more heads lead to more varied attention patterns. With a single head, the diversity score is 0 by definition since there's nothing to compare. With 2 heads, the model can already capture distinct patterns. As we increase to 4 and 8 heads, diversity continues to grow, though the incremental gains diminish. This suggests that beyond a certain point, additional heads may begin to learn redundant patterns.