Attention Analysis Limitations › Layer Selection and the Depth Problem

Even within a single head, the choice of layer matters enormously for interpretation. Early layers in BERT tend to encode local syntactic patterns. Middle layers encode semantic information. Late layers encode task-specific representations. An attention pattern at layer 1 and an attention pattern at layer 12 capture fundamentally different aspects of the input, even if they involve the same tokens.

The layer-by-layer specialization is not a fixed property of all transformers but emerges from the interaction of the training objective, the architecture, and the training data. Different models develop different layer specialization profiles depending on their pretraining setup. BERT, which uses masked language modeling, tends to develop syntactic representations in early layers and semantic representations in later layers. Causal language models like GPT may develop different layer structures because their training objective emphasizes different aspects of linguistic knowledge.

This depth problem means that any attention visualization is implicitly making a choice about which layer to report. Papers that report "the attention weights" without specifying layer and head selection methodology are presenting a cherry-picked view, whether intentionally or not. When attention heatmaps appear in papers without explicit methodology for layer and head selection, the reader should ask: how many layers and heads were examined before choosing this one? What would the heatmaps look like for the other layers?

The Attention Rollout and Attention Flow methods attempt to address this by propagating attention across layers, but they introduce their own assumptions about how information flows and whether attention at each layer can be straightforwardly multiplied. The assumption that layer-wise attention matrices compose multiplicatively is an approximation that may be more or less accurate depending on the specific model and input.

Rather than trying to read linguistic structure from attention patterns, an alternative approach is to use probing classifiers to detect what information is present in the hidden states at each layer. A probing classifier is a small linear model trained on top of a layer's hidden states to predict some linguistic property (part-of-speech tag, syntactic dependency role, named entity label, etc.). If the probing classifier achieves high accuracy, the hidden state contains information sufficient to determine that property, even if that information is not visible in the attention weights.

Probing provides a complementary perspective to attention analysis. Attention shows how information was routed; probing shows what information is present in the representations after routing. A thorough analysis combines both: if layer 6's hidden states contain high-accuracy predictions of subject-verb agreement (probing result) and head 4 at layer 6 shows strong subject-to-verb attention (attention result), that convergence provides better evidence for the claim that the model has learned subject-verb agreement than either method alone.