Multi-Head Attention › Multi-Head Attention

In the previous chapters, we built up the complete picture of scaled dot-product attention: how queries match with keys, how values get aggregated, and how masking controls information flow. A single attention mechanism can capture one type of relationship between tokens. But language is rich with multiple simultaneous relationships. When reading "The cat sat on the mat because it was tired," you simultaneously track syntactic structure (subject-verb agreement), semantic relationships (what "it" refers to), and positional patterns (nearby words). Can we design attention to capture all of these at once?

Multi-head attention answers this question by running multiple attention operations in parallel, each with its own learned projections. Instead of one attention mechanism looking for one type of pattern, we have several "heads," each specializing in different aspects of the input. This architectural choice is central to transformer performance and appears in every major language model from BERT to GPT-4.

Think of multi-head attention as a team of specialized analysts reviewing the same document simultaneously. One analyst focuses on grammatical structure, flagging every subject-verb pair. Another scans for semantic similarity, grouping words with overlapping meanings. A third tracks discourse references, connecting every pronoun to its antecedent. When they pool their reports, the combined picture is far richer than any single analyst could produce alone. Multi-head attention works the same way: multiple learned "views" of the same input, each detecting a different kind of relationship, combined into one unified representation.

The architecture achieves this without increasing the fundamental parameter budget or computational cost of attention. By splitting the model dimension across heads, each head operates on a lower-dimensional slice of the representation. The total work done is roughly the same as running one large attention operation, but the outputs capture far more structure. This parameter efficiency is one reason multi-head attention became the standard building block for transformers rather than an expensive luxury.

Multi-head attention does not explicitly program any head to detect syntax or coreference. Those specializations emerge through gradient descent as the model learns to minimize its prediction loss. The heads discover what is useful from data alone, which means they can adapt to whatever patterns matter most for a given language and task. This learned specialization, arising without explicit supervision, is one of the more remarkable properties of the mechanism.

In this chapter we will build multi-head attention from first principles, working through the mathematics carefully, implementing it in NumPy, and then examining how different heads develop distinct behaviors. We will also look at how practitioners configure and scale multi-head attention in real systems, from the original 512-dimensional transformer to models with tens of billions of parameters.