Attention Analysis Limitations › The Aggregation Problem in Multi-Head Attention

Beyond the single-head divergence, multi-head attention introduces additional complications for interpretation. A transformer with $H$ attention heads produces $H$ distinct attention patterns per layer, and a transformer with $L$ layers produces $L \times H$ total attention matrices. For BERT-base, that is $12 \times 12 = 144$ attention matrices per input. For larger models like GPT-3 or LLaMA-2, the number of attention matrices per input reaches into the thousands.

How do you summarize 144 matrices into one explanation? Common approaches include averaging across heads, taking the maximum, or selecting specific heads known to capture particular linguistic phenomena. Each choice makes implicit assumptions that affect the resulting interpretation, and none of these choices has strong theoretical justification.

Averaging across heads is mathematically convenient but conceptually problematic. Different heads often specialize in qualitatively different patterns: some track syntactic dependencies, others resolve coreference, and others attend to local context. Averaging conflates these distinct functions into a single blob that may not accurately represent any individual mechanism. If head 4 at layer 7 tracks subject-verb agreement and head 9 at layer 7 tracks local phrase boundaries, averaging them produces a pattern that does not clearly represent either function.

Selecting specific "interpretable" heads is better in principle but problematic in practice, because the mapping between attention patterns and linguistic functions is learned jointly with everything else. A head that attends to preceding tokens in one layer is not necessarily doing what you think it is, because its key and value projections interact with all other layers in complex ways. The interpretability of a specific head's pattern depends on what the surrounding computation is doing, and that context is invisible when you examine the head in isolation.

The head specialization literature, including the work of Voita et al. (2019) on pruning attention heads, has shown that many heads can be removed with minimal performance degradation. This is evidence against the interpretability of any individual head's attention pattern: if the head can be removed without changing behavior, its attention pattern cannot be the explanation for that behavior. The heads that survive pruning tend to be the ones performing computations that cannot be easily redundantly computed elsewhere, which are often the hardest to interpret because they implement the most complex transformations.

More recent work in mechanistic interpretability has revealed that individual attention heads can participate in multiple computational circuits simultaneously. A single head might contribute to both subject-verb agreement and long-range coreference resolution, depending on the input. Its attention pattern in any given forward pass reflects a superposition of these different functional roles, with each role weighted by its relevance to the current input.

This superposition makes it impossible to assign a fixed interpretation to a head's attention pattern. When you look at an attention heatmap for a specific input, you are seeing the combined output of all the circuits that head participates in, weighted by their activation on that input. The pattern is an aggregate of multiple functional roles, not a clean signal for any one of them.

The only way to disentangle these superimposed roles is to examine the head's behavior across many inputs, ideally inputs designed to selectively activate one circuit at a time. This is the approach taken by mechanistic interpretability research, which uses careful input design and activation patching to isolate individual circuits. We will explore these techniques in the chapter on mechanistic interpretability.