Attention Visualization › Summary

Attention visualization extracts the attention weight matrices produced during a forward pass and displays them as heatmaps or arc diagrams. Each transformer layer has multiple heads, each producing an independent $n \times n$ attention matrix for a sequence of $n$ tokens.

Extracting attention weights requires the output_attentions=True flag in Hugging Face models. The resulting tuple contains one tensor per layer, with shape (batch, heads, seq_len, seq_len). Individual heads can be sliced out and visualized directly.

Common patterns in attention heads include positional attention (attending to adjacent tokens), syntactic heads (tracking grammatical relations), coreference heads (linking pronouns to antecedents), and separator-attending heads. Entropy provides a quantitative measure of how focused a head's attention is, with low-entropy heads tending to perform more interpretable, specialized functions.

Attention weights are not causal explanations. High attention weight does not imply high causal influence on the output. Heads may not be modular, and the same head may show very different patterns on different inputs. Visualization tools like BertViz can mislead when they average across heads or layers without accounting for the fundamental incomparability of attention weights across depths.

The layer-wise progression of attention matters for interpretation. Early layers show positional and local patterns, middle layers encode syntactic structure, and late layers focus on semantically relevant tokens. Encoder-only models like BERT have fully populated attention matrices, while decoder-only models like GPT have lower-triangular matrices due to causal masking.

Residual connections carry information between layers independent of what any single attention head routes, which means single-layer heatmaps capture only part of the information routing story. Attention rollout provides a more complete picture by propagating attention weights backward through all layers while accounting for the residual stream.

These limitations motivate the more rigorous techniques covered in Attention Analysis Limitations and the broader interpretability toolkit developed in subsequent chapters. Attention visualization remains useful as a quick hypothesis-generation step and for communication with non-expert audiences, but it is now understood as one tool among many rather than a sufficient analysis on its own. A structured workflow that combines entropy analysis, multi-example examination, specific hypothesis testing, and cross-validation with gradient-based methods produces more reliable conclusions than isolated heatmap inspection.