Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Attention Visualization › Attention Head Specialization

Michael BrenndoerferFebruary 11, 20267 min read

Language AI HandbookAll chapters

Part 59 of 67Interpretability

Chapter 447Attention Visualization › Attention Head Specialization

Reading section 5 of 15 · Attention Head Specialization · 7 minSection 5 of 15 · 7 min

Section 5 of 15 · 7 min read

Attention Visualization › Attention Head SpecializationOne section · Sections in this chapter5 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of the most striking findings from early attention analysis is that individual heads often exhibit recognizable functional patterns. Researchers at Hugging Face and elsewhere systematically analyzed attention heads in BERT and found that certain heads reliably perform specific linguistic roles.

Common Head PatternsLink Copied

Through careful examination of many sentences, several recurring head types have been documented. These patterns are not accidental: they reflect how the model solved the masked language modeling objective, which requires understanding contextual meaning, syntactic agreement, and semantic relationships across a sentence.

Positional heads attend predominantly to the previous or next token, implementing a form of local context window. These heads appear consistently and are easy to spot: the heatmap shows a band just above or below the diagonal. The previous-token head is particularly common in early layers, where it serves a function analogous to a left-context n-gram window.

Syntactic heads track grammatical relationships. For example, some heads connect verbs to their subjects or objects, or link determiners to the nouns they modify. When you visualize these heads on sentences with clear syntactic structure, the attention matrix looks almost like a dependency parse tree: non-zero cells correspond to syntactic arcs. Clark et al. (2019) identified specific BERT heads that align well with dependency parse relations at rates significantly above chance, particularly for relations like nominal subjects, direct objects, and adjectival modifiers.

Coreference heads link pronouns to their antecedents. In a sentence like "The lawyer questioned the witness. She doubted his testimony," a coreference head in a later layer might show strong attention from "She" toward "The lawyer" and from "his" toward "the witness." These heads are particularly interesting because coreference is a long-range phenomenon that requires integrating information across sentence boundaries. The fact that some heads specialize in this suggests that even within-sentence attention can partially solve cross-sentence reference tasks when the context fits within the model's maximum sequence length.

Separator-attending heads show strong attention toward the [SEP] token, particularly in later layers. These heads are thought to implement a form of "no-op" or soft reset, attending to a semantically neutral token when no other position is relevant. The [SEP] head is a kind of "I have nothing useful to look at" signal that the model can route to when the query position has already gathered the context it needs. This pattern is specific to BERT-style models trained with sentence-pair tasks; GPT-style models do not have [SEP] tokens and therefore cannot develop this behavior.

Self-attending heads have a strong diagonal, meaning each token attends mostly to itself. These appear to aggregate local information without changing the representation significantly. They are most prevalent in very early layers and may serve an initialization function before the model builds richer cross-token representations.

The existence of these patterns provided early hope that transformers were learning interpretable linguistic structure rather than performing opaque statistical computation. However, the relationship between attention patterns and linguistic roles is correlational rather than causal. A head that looks like it tracks subjects need not be computing subject identification in any meaningful sense. The weights might have converged to this pattern for other reasons that happen to produce similar-looking attention matrices.

The Specialization Discovery ProcessLink Copied

How did researchers discover these patterns in the first place? The methodology matters, because it affects how confident you should be in the findings.

The most common approach is qualitative: visualize many attention heads on many sentences and look for patterns that recur. When head X in layer Y consistently shows strong attention from pronouns to nearby nouns, you hypothesize that it tracks coreference. This is hypothesis generation, not hypothesis testing.

A more rigorous approach is correlation analysis: probe whether the attention pattern aligns quantitatively with a linguistic annotation. For a syntactic dependency head, you would take a parsed corpus, extract the dependency arcs, and measure what fraction of the model's attention edges match gold parse arcs. If the match rate is significantly above the random baseline, you have quantitative evidence for the hypothesis. Clark et al. (2019) used exactly this approach and found several BERT heads with dependency alignment scores above 0.5, meaning they matched the gold parse arc more than half the time across diverse sentences.

The limitation of correlation analysis is that it tells you whether the head's pattern resembles a linguistic structure, not whether the model is using that structure for its predictions. A head can produce syntactically organized attention without the downstream layers caring about those attention weights. This distinction motivates more interventional tools like probing classifiers and activation patching, covered in later chapters.

Attention Entropy as a Specialization MeasureLink Copied

One quantitative way to characterize heads is through the entropy of their attention distributions. A head with low entropy has sharp, focused attention concentrated on a few tokens. A head with high entropy distributes attention broadly. Plotting entropy across layers reveals structural patterns in how information flows through the model.

Entropy for a single attention row is:

H(A_i) = -\sum_{j=1}^{n} A_{ij} \log A_{ij}

where:

$A_i$ : the $i$ -th row of the attention matrix, representing the attention distribution for token $i$
$A_{ij}$ : the attention weight from token $i$ to token $j$
The sum runs over all $n$ positions in the sequence

Higher entropy means more diffuse attention; lower entropy means more focused attention on specific positions.

To interpret this concretely: if a sequence has 14 tokens and a head distributes attention perfectly uniformly, each weight is $1/14$ and the entropy is $\log(14) \approx 2.64$ nats. A head that concentrates all attention on a single token has entropy $0$ . Real heads fall between these extremes, but syntactic and coreference heads typically have much lower entropy than sentence-level aggregation heads.

The entropy measure is useful precisely because it is aggregatable. You can compute mean entropy across all query positions in a sentence, then average across sentences, to get a single number per head that characterizes how selective that head is across the whole dataset. This makes it possible to rank all 144 BERT-base heads from most focused to most diffuse and focus your visual inspection on the most interesting candidates.

In[11]:

Code

import numpy as np


def attention_entropy(attn_matrix):
    """Compute mean entropy across all query positions for one attention head."""
    # Clip to avoid log(0)
    attn_clipped = np.clip(attn_matrix, 1e-9, 1.0)
    entropy_per_row = -np.sum(attn_clipped * np.log(attn_clipped), axis=1)
    return float(np.mean(entropy_per_row))


# Compute entropy for all heads across all layers
entropy_matrix = np.zeros((num_layers, num_heads))
for layer in range(num_layers):
    for head in range(num_heads):
        attn = attention_weights[layer][0, head].numpy()
        entropy_matrix[layer, head] = attention_entropy(attn)

Out[12]:

Visualization

Heatmap of attention entropy values across 12 layers and 12 heads of BERT-base — Mean attention entropy per head across all 12 layers of BERT-base, with each cell representing one attention head. Lower entropy (darker blue) indicates focused, selective attention concentrated on few tokens, while higher entropy (lighter blue) corresponds to diffuse attention spread broadly. Early layers tend to show more variable entropy, while later layers often develop more specialized low-entropy heads alongside high-entropy aggregate heads.

Out[13]:

Console

Most focused head: Layer 3, Head 1 (entropy = 0.024)
Most diffuse head: Layer 1, Head 1 (entropy = 2.540)

The entropy profile often shows that early layers have relatively high-entropy heads (diffuse attention gathering broad context), while later layers develop more specialized, low-entropy heads. However, this pattern varies substantially across sentences, models, and fine-tuning tasks.