Multi-Head Attention › Summary

Multi-head attention extends single-head attention by running multiple attention operations in parallel, each with its own learned projections. This architectural choice significantly impacts model capacity and performance.

Key takeaways from this chapter:

• Parallel heads: Multi-head attention computes $h$ independent attention operations, each learning different relationship patterns between tokens. Each head can specialize in detecting specific linguistic phenomena.
• Dimension splitting: The model dimension $d_{\text{model}}$ is divided among heads, with each head operating on dimension $d_k = d_{\text{model}} / h$. This keeps total computation comparable to single-head attention while enabling multiple perspectives.
• Four-step process: (1) Project input to Q, K, V for each head using learned matrices $\mathbf{W}^Q_i$, $\mathbf{W}^K_i$, $\mathbf{W}^V_i$; (2) compute scaled dot-product attention independently per head; (3) concatenate head outputs; (4) apply final output projection $\mathbf{W}^O$.
• Head specialization: Different heads learn to focus on different aspects: positional patterns, syntactic relationships, semantic similarity, or coreference links. This emerges from training without explicit supervision.
• Parameter efficiency: The total parameter count is $4 \times d_{\text{model}}^2$, similar to single-head attention with an output projection. The gain is in representational diversity, not parameter count.
• Practical implementations: Efficient implementations batch all heads into single tensor operations, using GPU parallelism through combined QKV projections.

In the next chapter, we'll examine the computational complexity of attention in detail. Understanding the $O(n^2)$ scaling (where $n$ is sequence length) and its implications is essential for working with long sequences and motivates the efficient attention variants used in modern long-context models.