Attention Analysis Limitations: When Weights Mislead

Transformers have given us something remarkable: a mechanism that, in principle, tells us exactly how much each input token influences each output token. Attention weights are explicit, inspectable, and visualizable. When you pass a sentence through BERT and display its attention heatmaps, you can watch the model "look at" different words. Researchers seized on this as a window into the black box, publishing papers with titles like "Are Sixteen Heads Really Better Than One?" and generating beautiful visualizations of linguistic structure. For a field that had long struggled to explain why neural networks made the decisions they did, attention seemed like a gift: interpretability baked into the architecture itself.

The problem is that attention weights do not reliably measure importance. The question of whether high attention means "this token matters" requires a qualified answer: sometimes yes, often no, and always more complicated than the heatmap suggests. Understanding why this is true, what researchers have done to test it, and what alternatives exist gives you a foundation for responsible model analysis. This chapter examines the harder question: what should you conclude from those visualizations? The short answer is: considerably less than many early papers assumed.

As we explored in the previous chapter on attention visualization, the mechanics of extracting and rendering attention weights are straightforward. The hard part is interpreting what those weights mean, and that is where many early papers went wrong. A heatmap is a data visualization, not an explanation. Treating it as one requires a set of assumptions about the relationship between attention weights and model behavior, and those assumptions frequently fail.

The stakes here are real. When practitioners use attention heatmaps to audit models for bias, or when researchers cite attention patterns as evidence for their claims about what a model has learned, they are making implicit claims about mechanism. If those claims are unsupported, the audit is unreliable and the evidence is misleading. The story of attention as explanation is a cautionary tale about the difference between measurement and understanding.

Before examining the limitations, it helps to be precise about what attention weights represent mathematically. In scaled dot-product attention, for each query token at position $i$, the model computes a distribution over all key positions $j$:

where:

• $\alpha_{ij}$ is the attention weight from position $i$ to position $j$
• $q_i$ is the query vector at position $i$
• $k_j$ is the key vector at position $j$
• $d_k$ is the dimension of the key vectors, used to scale the dot products and prevent gradient vanishing in softmax

The output at position $i$ is the weighted sum of value vectors:

where $v_j$ is the value vector at position $j$.

The attention weight $\alpha_{ij}$ quantifies how much position $i$'s query "matches" position $j$'s key. It determines the mixing coefficients for value vectors. But note what it does not measure: it does not measure the causal effect of token $j$ on the model's final prediction. The value vectors $v_j$ are themselves transformations of the input, and different value vectors can have wildly different norms, meaning that a small attention weight to a token with a large-norm value vector might dominate a large attention weight to a token with a near-zero value vector.

This distinction between attention and importance is not merely theoretical. The final influence of a token on the model output depends on both the attention weight and the value vector. Two tokens can receive equal attention weights but have vastly different impacts if their value vectors differ in magnitude. Conversely, a token with low attention but a large-norm value vector might matter far more than the heatmap suggests.

To see why this happens, consider what value vectors represent. The key and query projections determine which tokens attend to which, but the value projection determines what information gets mixed in when attention occurs. A model trained on sentiment analysis might learn to give the value projection for sentiment-bearing tokens (words like "terrible," "excellent," "not") large norms, because these tokens carry the semantically decisive information. Meanwhile, syntactic anchor tokens like articles and prepositions might accumulate high attention weights simply because the attention mechanism uses them as positional landmarks, but their value projections carry little information about the prediction.

This is a fundamental design property of the transformer architecture. Attention weights and value vectors are independent parameters, both learned jointly. Nothing in the training objective forces a correspondence between the magnitude of attention and the magnitude of downstream effect. The model is free to learn configurations where these diverge dramatically, and in practice it often does. The model's goal is to minimize task loss, not to make attention weights interpretable.

The practical consequence is that attention-based heatmaps can be systematically misleading in exactly the settings where we most want interpretability. When a model is processing sentiment, we want to know whether it is tracking negation ("not good") versus intensifiers ("really good") correctly. Attention might show the model attending to function words for positional grounding while the actual semantic content flows through value vectors with high norms on the content words. The heatmap looks like syntactic structure; the actual computation is semantic.

There is a second architectural reason why attention weights underspecify information flow: the residual stream. In each transformer block, the output of the attention layer is added to its input through a residual connection before being passed to the feed-forward layer. This means that every token's representation at every layer contains a mix of information from the current layer's attention computation and all previous layers' outputs.

Information in the residual stream can bypass the attention mechanism entirely. A token's representation can carry task-relevant information forward from earlier layers without that information showing up in the attention weights of later layers. When you observe a late layer's attention pattern, you are seeing how that layer routes information given all the prior computation that has already been incorporated. The attention weights at layer 10 do not tell you what information was established at layer 2.

This is why zeroing out attention weights for specific tokens does not always change the model's behavior in the expected way. The model may have already encoded the necessary information in the residual stream from earlier layers, making the current layer's attention to that token redundant. Brunner et al. (2020) demonstrated exactly this: transformers can "hide" information by encoding it in residual stream representations that bypass attention, allowing the model to maintain performance even when specific attention connections are removed.

The limitations of attention as explanation were crystallized in the landmark paper by Jain and Wallace (2019): "Attention is not Explanation." Their work set off a productive but contested debate that reshaped how the NLP community thinks about interpretability. Before this paper, it was common practice to include attention heatmaps in papers as informal evidence of model behavior, with the implicit assumption that high-attention tokens were somehow responsible for the model's predictions.

Jain and Wallace examined attention-based text classification models and asked a simple question: if you change the attention weights, does the model's prediction change? Their experiments revealed two troubling findings.

The first finding: attention does not correlate with gradient-based importance. For many inputs, tokens that received high attention were not the same tokens that, when perturbed, most changed the prediction. The gradient-based measure (which does capture causal importance) and the attention-based measure often disagreed substantially. In some experiments, the Kendall rank correlation between attention and gradient importance was close to zero, indicating near-random agreement about which tokens mattered.

The second finding: alternative attention distributions can produce identical predictions. For a given input, they could find many different attention distributions that produced the same model output and the same prediction. This means attention is underdetermined: many different "explanations" (different attention patterns) are consistent with the same behavior. If attention were truly the mechanism driving the decision, such degeneracy would be impossible. You cannot claim that attention pattern A explains the prediction if attention pattern B, which looks completely different, produces the same prediction.

The practical implication is stark. If you can freely modify attention weights without changing the model's behavior, then attention weights are not reliably explaining the model's behavior. They may be a byproduct of the computation rather than the driver. A car's speedometer can be swapped for a different model without affecting the engine's operation, but we would not say the speedometer caused the car to move.

Wiegreffe and Pinter (2019) responded with "Attention is not not Explanation," arguing that Jain and Wallace's adversarial attention construction required training a separate model and therefore did not truly show that the original attention was uninformative. Their counter-argument: the existence of alternative attention distributions that produce the same prediction does not mean the original attention is not a faithful explanation, because there may be many faithful explanations.

The key distinction they drew is between the existence of alternative explanations and the faithfulness of the original. A faithful explanation accurately describes how the model computed its output. The fact that other computations could have reached the same output does not make the original computation unfaithful, any more than the existence of multiple routes to a destination means the route you took was not the one you drove.

They proposed a stricter criterion: is it possible to train a model that achieves similar performance with a completely different attention distribution? Using this criterion, they found that in many tasks, attention weights are more constrained than Jain and Wallace suggested. When the task requires routing specific information through attention (rather than through feed-forward or residual mechanisms), the attention patterns become less exchangeable.

The debate matters because it reveals that the notion of "explanation" is ambiguous. There are at least three different things you might want from an attention explanation:

• Faithfulness: Does the explanation accurately describe how the model computed its output?
• Plausibility: Does the explanation match human intuitions about which tokens are important?
• Sufficiency: Is the explanation sufficient to reproduce the model's behavior?

Attention often scores reasonably on plausibility but poorly on faithfulness and sufficiency. We naturally read attention heatmaps and think "yes, those are the relevant words," but this human plausibility does not imply the model is using them in the way we imagine. The alignment between human intuition and model attention is partly a coincidence of training data distribution, not evidence of mechanistic correspondence.

The gap between plausibility and faithfulness is worth dwelling on, because it explains why attention visualizations are so seductive and so dangerous. Humans evaluate natural language explanations using the same prior knowledge they bring to reading text. If a model attends to "bank" in the sentence "The bank was full of fish" and the model predicted a river-related sense of "bank," the attention to "fish" looks plausible because fish do swim in rivers. But the model might have reached this conclusion through the value vector for "fish" having been pushed toward river-sense representations during pretraining, with the attention to "fish" merely serving as a positional anchor rather than a disambiguating signal.

The plausibility check passes because humans and the model share background knowledge. The faithfulness check would require verifying that removing or replacing "fish" changes the prediction, which might not be the case if the river-sense representation was already established earlier in the forward pass.

This is a general pattern in interpretability research: explanations that align with human intuition are not necessarily mechanistically accurate. They may satisfy our intuitions by accident, exploiting the correlation between what we think matters and what the model has learned, without tracing the causal pathway of computation.

Subsequent work by Brunner et al. (2020) provided additional evidence against naive attention interpretability. They showed that transformers can learn to "hide" information by encoding it in residual stream representations that bypass attention. Even if you zero out certain attention connections, the model can recover information through skip connections and layer normalization.

This points to a fundamental architectural issue: attention is not the only information pathway in a transformer. The residual stream passes information directly from layer to layer. Feed-forward layers can store and retrieve factual knowledge independently of attention patterns. Attention is one mechanism among several, and measuring it alone gives an incomplete picture of information flow. The feed-forward layers in particular have been shown to function as key-value memories that store factual associations, and this storage is entirely invisible to attention analysis.

The implication for interpretability research is that any explanation method based solely on attention is necessarily incomplete. It is like trying to understand a program by only reading the function call stack and ignoring the global variables and in-place mutations. You might get a partial picture, but you will systematically miss everything that happens outside the scope of what you measured.

Because attention weights reflect mixing coefficients rather than causal importance, interpretability researchers have developed gradient-based methods that more directly measure how the output changes when inputs are perturbed. These methods approach the question differently: instead of asking "where did the model look?", they ask "which tokens, if changed, would most change the model's decision?"

The simplest gradient-based method computes how sensitive the output is to each input token embedding. Given a model output $y$ and input embedding $e_i$ for token $i$, the saliency score is:

where:

• $s_i$ is the saliency score for token $i$
• $y$ is the model's scalar output (e.g., the logit for the predicted class)
• $e_i$ is the embedding vector for token $i$
• $\|\cdot\|$ denotes the $\ell_2$ norm of the gradient vector

Gradient sensitivity provides the intuition: if slightly changing token $i$'s embedding substantially changes the output, then token $i$ has high causal importance. This is a direct measure of sensitivity, grounded in calculus rather than attention mechanics. If the gradient is large at a token, perturbing that token would change the output substantially, which is precisely what we mean when we say a token is "important" for the prediction.

Input gradient saliency has its own limitations. Gradients can be noisy and can saturate in regions of the input space where the model is highly confident. A gradient near zero might mean the token is unimportant, or it might mean the model is in a flat region where many perturbations produce the same prediction, not because the token lacks semantic content but because the model has already committed to a decision with high confidence. This saturation problem is particularly acute for deep networks where the loss surface has many flat regions.

Another issue with raw input gradients is instability. The gradient at a single point may not be representative of the token's overall importance, because the gradient measures local sensitivity rather than global influence. If you perturb the embedding slightly, the gradient at the new point may look quite different from the gradient at the original point.

Integrated Gradients (Sundararajan et al., 2017) addresses the saturation problem by integrating the gradient along a path from a baseline input (typically a zero embedding or a mask token) to the actual input:

where:

• $e_i$ is the actual embedding of token $i$
• $e'_i$ is the baseline embedding (often zeros)
• $F(\cdot)$ is the model's output function
• $\alpha$ is a scalar interpolation parameter ranging from 0 to 1
• The integral accumulates gradient information across the full interpolation path from baseline to actual input

Integrated Gradients satisfies an important axiom called "completeness": the sum of attributions across all inputs equals the difference in model output between the actual input and the baseline. Formally:

This gives the attributions a consistent interpretation and prevents the method from attributing importance that does not exist. If the model's output is 0.9 for the actual input and 0.1 for the baseline, the attributions must sum to 0.8, accounting for the full difference in prediction.

The practical advantage over simple input gradients is robustness to saturation. By integrating over the path, you capture gradient information from regions where the model is less confident, including the interpolated inputs where the model is transitioning from its baseline state to its actual prediction. This avoids the zero-gradient trap where a confident model appears to be insensitive to all inputs.

The choice of baseline matters significantly for the interpretation. Using a zero embedding baseline treats the "absence" of a token as the reference point, measuring how much each token contributes relative to having no input. Using a mask token baseline treats the masked-out token as the reference point, measuring how much information each specific token provides beyond what the model would predict with unknown inputs. These give different attributions and both are defensible, depending on what question you are trying to answer.

Abnar and Zuidema (2020) proposed methods that attempt to fix attention-based attribution rather than abandon it. Their observation: because transformers stack multiple attention layers, the attention at layer $L$ is not directly connected to the input. Information flows through all intermediate layers, and examining attention at a single layer misses how that information was aggregated across the full depth of the network.

Attention Rollout computes the effective attention from output to input by multiplying attention matrices across layers:

where $A^{(l)}$ is the attention matrix at layer $l$. Because each layer also has a residual connection, they add an identity matrix to account for the direct pass-through:

where $I$ is the identity matrix. The rollout then uses these adjusted matrices. The resulting $A_\text{roll}$ estimates how much each input token contributes to each output position across the full model depth. This is a significant improvement over single-layer attention because it accounts for how attention patterns compose across layers.

Attention Flow further improves on Attention Rollout by treating the attention graph as a flow network and computing maximum flow rather than simple matrix products, which better handles the normalization properties of attention. In a flow network, information can travel along multiple paths, and the maximum flow captures the bottleneck capacity of each path. This is more faithful to how information propagates through the transformer, where a single token might contribute to an output through several different intermediate tokens rather than through a single direct attention path.

Both methods have their own limitations. Attention Rollout assumes that attention matrices at different layers can be validly multiplied, which requires that the value vectors at each layer have similar properties. Attention Flow is more principled but computationally expensive and harder to scale to large models. Neither method fully accounts for the role of feed-forward layers and residual connections in information storage and retrieval.

A pragmatic hybrid approach is gradient-weighted attention. Rather than using raw attention weights, you weight each attention head's output by the gradient of the loss with respect to that head's output:

where:

• $\tilde{\alpha}_{ij}^{(h)}$ is the gradient-weighted attention from position $i$ to $j$ in head $h$
• $\alpha_{ij}^{(h)}$ is the raw attention weight
• The gradient term measures how much the loss changes when you perturb that specific attention weight

This hybrid retains the spatial structure of attention (which tokens attend to which) while incorporating gradient information about which of those attention connections matter for the prediction. It tends to outperform raw attention visualization as an explanation method because it filters out attention connections that are high-weight but causally inert, keeping only the connections where both the attention weight is substantial and the gradient confirms that connection contributes to the prediction.

The intuition behind gradient-weighted attention is similar to Grad-CAM in computer vision, which weights spatial feature maps by their gradient with respect to the class score to produce localization maps. In both cases, the spatial structure (attention weights or feature maps) is refined by gradient information to produce a more causally grounded attribution.

SHapley Additive exPlanations (SHAP) values provide yet another approach to token attribution, grounded in cooperative game theory. The Shapley value of a token measures its marginal contribution to the model's prediction, averaged over all possible subsets of tokens. Given a model $F$ and a set $S$ of tokens, the Shapley value of token $i$ is:

where $T$ is the full set of tokens. This computes a weighted average of the marginal contribution of token $i$ across all possible subsets of the other tokens. The weighting ensures that the Shapley values satisfy several fairness properties, including efficiency (the values sum to the model's output) and symmetry (tokens that contribute equally receive equal values).

SHAP values are computationally expensive to compute exactly for transformer models because the number of subsets is exponential in the number of tokens. In practice, approximations are used, including KernelSHAP (which approximates Shapley values using a weighted linear model) and TreeSHAP (which is exact for tree-based models but requires approximations for neural networks). These approximations introduce their own assumptions and potential inaccuracies.

Despite the computational expense, SHAP values have attractive properties for interpretability. They are model-agnostic (the same method applies regardless of architecture), they satisfy consistency and fairness axioms, and they produce attributions that sum to the model's output. When comparing across different models or architectures, SHAP values provide a common currency for attribution that attention weights cannot provide because attention is architecture-specific.

To make the divergence concrete, let's walk through a small example. Suppose a sentiment classifier processes the sentence: "The movie was not terrible, it was genuinely awful."

If we extract the attention weights from a specific head at a specific layer, we might see high attention to "movie" and "was" from the [CLS] token, because these are common syntactic anchors. The words "terrible" and "awful" might receive moderate attention. Syntactic anchors attract attention not because they are semantically decisive but because they serve as positional landmarks that help the model orient itself within the sentence structure. Function words and common nouns are often the most predictable tokens in context, making them useful attention targets for structural grounding.

But if we compute input gradients, "not" and "awful" typically receive the highest scores, because flipping the sentiment requires changing these specific tokens. The word "was" is irrelevant to sentiment: you could replace it with "seemed" and the prediction stays identical. The gradient correctly identifies "not" as high-importance because negation is what transforms the sentence from a negative-without-hedge ("the movie was terrible") to a double-negation with different valence ("it was not terrible"). The gradient for "awful" is high because it is the strongest sentiment word in the sentence, whose replacement with a neutral noun would dramatically change the prediction.

This is the practical failure mode. A researcher looking only at attention might conclude "the model is attending to 'movie' and 'was,'" which sounds like syntactic structure learning. A researcher looking at gradients concludes "the model is sensitive to 'not' and 'awful,'" which correctly identifies the semantically relevant negation and intensifier. These are different stories, and the gradient story is the more faithful one. The attention story is not wrong per se (the model does attend to those tokens), but it answers a different question than the one the researcher usually has in mind.

The same divergence appears in question answering tasks, where attention frequently distributes across the question and context tokens for structural grounding, while gradients identify the specific evidence tokens whose content determines the answer. In machine translation, attention often aligns with grammatical boundaries while gradients show the content words that most constrain the target translation. The pattern is consistent: attention captures routing structure; gradients capture semantic influence.

We implement a self-contained demonstration using NumPy to simulate the core divergence between attention weights and gradient-based importance. The key phenomenon, that attention routing does not predict gradient sensitivity, emerges from a concrete property: different tokens have value vectors of different norms, so a small attention weight multiplied by a large-norm value vector can contribute more to the output than a large attention weight multiplied by a small-norm value vector.

We set up the token sequence and define simulated attention weights and value vector norms to mirror what research has found in real transformer models.

Beyond the single-head divergence, multi-head attention introduces additional complications for interpretation. A transformer with $H$ attention heads produces $H$ distinct attention patterns per layer, and a transformer with $L$ layers produces $L \times H$ total attention matrices. For BERT-base, that is $12 \times 12 = 144$ attention matrices per input. For larger models like GPT-3 or LLaMA-2, the number of attention matrices per input reaches into the thousands.

How do you summarize 144 matrices into one explanation? Common approaches include averaging across heads, taking the maximum, or selecting specific heads known to capture particular linguistic phenomena. Each choice makes implicit assumptions that affect the resulting interpretation, and none of these choices has strong theoretical justification.

Averaging across heads is mathematically convenient but conceptually problematic. Different heads often specialize in qualitatively different patterns: some track syntactic dependencies, others resolve coreference, and others attend to local context. Averaging conflates these distinct functions into a single blob that may not accurately represent any individual mechanism. If head 4 at layer 7 tracks subject-verb agreement and head 9 at layer 7 tracks local phrase boundaries, averaging them produces a pattern that does not clearly represent either function.

Selecting specific "interpretable" heads is better in principle but problematic in practice, because the mapping between attention patterns and linguistic functions is learned jointly with everything else. A head that attends to preceding tokens in one layer is not necessarily doing what you think it is, because its key and value projections interact with all other layers in complex ways. The interpretability of a specific head's pattern depends on what the surrounding computation is doing, and that context is invisible when you examine the head in isolation.

The head specialization literature, including the work of Voita et al. (2019) on pruning attention heads, has shown that many heads can be removed with minimal performance degradation. This is evidence against the interpretability of any individual head's attention pattern: if the head can be removed without changing behavior, its attention pattern cannot be the explanation for that behavior. The heads that survive pruning tend to be the ones performing computations that cannot be easily redundantly computed elsewhere, which are often the hardest to interpret because they implement the most complex transformations.

More recent work in mechanistic interpretability has revealed that individual attention heads can participate in multiple computational circuits simultaneously. A single head might contribute to both subject-verb agreement and long-range coreference resolution, depending on the input. Its attention pattern in any given forward pass reflects a superposition of these different functional roles, with each role weighted by its relevance to the current input.

This superposition makes it impossible to assign a fixed interpretation to a head's attention pattern. When you look at an attention heatmap for a specific input, you are seeing the combined output of all the circuits that head participates in, weighted by their activation on that input. The pattern is an aggregate of multiple functional roles, not a clean signal for any one of them.

The only way to disentangle these superimposed roles is to examine the head's behavior across many inputs, ideally inputs designed to selectively activate one circuit at a time. This is the approach taken by mechanistic interpretability research, which uses careful input design and activation patching to isolate individual circuits. We will explore these techniques in the chapter on mechanistic interpretability.

Even within a single head, the choice of layer matters enormously for interpretation. Early layers in BERT tend to encode local syntactic patterns. Middle layers encode semantic information. Late layers encode task-specific representations. An attention pattern at layer 1 and an attention pattern at layer 12 capture fundamentally different aspects of the input, even if they involve the same tokens.

The layer-by-layer specialization is not a fixed property of all transformers but emerges from the interaction of the training objective, the architecture, and the training data. Different models develop different layer specialization profiles depending on their pretraining setup. BERT, which uses masked language modeling, tends to develop syntactic representations in early layers and semantic representations in later layers. Causal language models like GPT may develop different layer structures because their training objective emphasizes different aspects of linguistic knowledge.

This depth problem means that any attention visualization is implicitly making a choice about which layer to report. Papers that report "the attention weights" without specifying layer and head selection methodology are presenting a cherry-picked view, whether intentionally or not. When attention heatmaps appear in papers without explicit methodology for layer and head selection, the reader should ask: how many layers and heads were examined before choosing this one? What would the heatmaps look like for the other layers?

The Attention Rollout and Attention Flow methods attempt to address this by propagating attention across layers, but they introduce their own assumptions about how information flows and whether attention at each layer can be straightforwardly multiplied. The assumption that layer-wise attention matrices compose multiplicatively is an approximation that may be more or less accurate depending on the specific model and input.

Rather than trying to read linguistic structure from attention patterns, an alternative approach is to use probing classifiers to detect what information is present in the hidden states at each layer. A probing classifier is a small linear model trained on top of a layer's hidden states to predict some linguistic property (part-of-speech tag, syntactic dependency role, named entity label, etc.). If the probing classifier achieves high accuracy, the hidden state contains information sufficient to determine that property, even if that information is not visible in the attention weights.

Probing provides a complementary perspective to attention analysis. Attention shows how information was routed; probing shows what information is present in the representations after routing. A thorough analysis combines both: if layer 6's hidden states contain high-accuracy predictions of subject-verb agreement (probing result) and head 4 at layer 6 shows strong subject-to-verb attention (attention result), that convergence provides better evidence for the claim that the model has learned subject-verb agreement than either method alone.

Given these limitations, it is useful to be explicit about the appropriate and inappropriate uses of attention visualization. The goal is not to abandon attention analysis entirely but to calibrate the strength of the claims you draw from it.

Attention weights are appropriate for:

• Exploratory analysis: Forming hypotheses about what the model might be doing, to be tested with more rigorous methods
• Debugging structural issues: If a model attends only to [CLS] and [SEP] tokens uniformly, something is likely wrong with the training setup
• Educational demonstration: Showing students what the attention mechanism looks like in practice, with appropriate caveats about interpretation
• Qualitative sanity checks: Verifying that a model attends to relevant tokens at all, not that attention weights represent precise importances
• Anomaly detection: Identifying unusual attention patterns that might indicate distribution shift, adversarial inputs, or unexpected behavior worth investigating further

Attention weights are inappropriate for:

• Causal claims: Concluding that token $j$ caused the prediction because it received high attention
• Contrastive explanation: Explaining why the model chose class A over class B based on attention patterns
• Faithfulness assessment: Assuming that high attention means the model's decision depends heavily on that token
• Model comparison: Comparing two models based on their attention patterns, since different attention configurations can implement the same function
• Bias audits: Concluding that a model is or is not biased based on whether it attends to demographic attributes in the input

The last point about bias audits deserves emphasis. Several papers have attempted to detect bias in transformer models by examining whether attention is drawn to demographic attributes (race, gender, nationality) in text. If a model attends to "she" when making predictions about a nurse but "he" when making predictions about a doctor, the argument goes, attention reveals gender bias. But this reasoning conflates attention with causal influence. The model might attend to pronouns for grammatical agreement while making its prediction based entirely on the occupation noun and its associations in pretraining data. The attention pattern would show the grammatical function, not the bias.

Despite the critique, there are contexts where attention analysis provides real value. The key is using it for the right questions, in architectures where the attention mechanism is constrained to carry specific, measurable information.

Induction heads are a case where attention analysis reveals a concrete mechanism. Work in mechanistic interpretability has shown that specific attention heads in transformer language models implement a precise copying operation: they attend to previous occurrences of a token in the context and copy the token that followed it. When you see a sequence like "Harry Potter, Harry", an induction head will attend strongly from the second "Harry" to the first "Harry" and use the value vector to retrieve "Potter" as the predicted continuation. These induction heads are directly observable in attention patterns, and the attention tells you something real about what the head is computing, because the computation is architecturally required to flow through attention.

Cross-attention in encoder-decoder models is another case where attention can be informative. When a translation model attends from the target language back to the source language, the attention pattern represents the alignment between source and target tokens. This is because the cross-attention mechanism is architecturally required to route information from encoder to decoder: there is no residual pathway that bypasses cross-attention for encoder-decoder communication. The attention to source tokens is therefore a necessary condition for using those tokens' representations, making the attention a more faithful proxy for information flow than it would be in self-attention where residual connections provide alternative pathways.

In summarization models, cross-attention over the source document can reveal which source sentences the model is drawing on when generating each summary sentence. While this is not a perfect faithfulness measure (the model might use multiple source sentences for structural scaffolding and only one for content), it provides useful signal about information sourcing that has practical value for understanding and debugging model behavior.

The common thread across these cases: attention is informative when the architecture constrains it to carry specific, measurable information, and when the specific computation you care about must pass through the attention bottleneck. When the architecture allows attention to "hide" information in value vectors, residual streams, or feed-forward layers, the attention weights become less reliable as explanations.

Even when individual attention weights lack causal interpretation, aggregate attention patterns across many examples can serve as behavioral signatures that distinguish different model behaviors. If you compare the attention patterns of a model that has been fine-tuned for toxicity detection versus a model fine-tuned for sentiment analysis, the systematic differences in how they attend to certain vocabulary classes can be informative about the different features each model has learned to use, without requiring each individual attention weight to be causally interpretable.

This use of attention as a behavioral fingerprint is quite different from using it as a causal explanation of individual predictions. It acknowledges that attention patterns are a noisy, indirect signal while still extracting information from the statistical regularities in that signal. The key is to examine patterns across many examples rather than drawing conclusions from single instances.

For practitioners doing actual model analysis, these findings suggest a layered approach to interpretability that uses the most appropriate tool for each question.

Start with gradient-based methods. Integrated Gradients or SHAP values provide more principled attribution scores with theoretical guarantees. They are slower to compute than attention but far more reliable. For task-critical applications like medical diagnosis support or legal reasoning assistance, the additional computation time is worthwhile to get attributions with stronger theoretical grounding.

Use attention for pattern exploration, not causal conclusions. If integrated gradients show a token and attention also shows it, that is meaningful convergent evidence. Two independent methods agreeing on the importance of a token provides stronger support than either method alone. If only attention shows it, treat it as a hypothesis to test with gradient-based methods.

Be specific about what you are measuring. "Attention weights from layer 6, head 4, averaged over the last 100 examples for the [CLS] token" is a meaningful quantity. "The model's attention" is not. The lack of specificity in how attention is extracted and aggregated is one of the most common sources of unreliable claims in interpretability research.

Compare methods. When attention and gradient-based methods disagree, take note. The disagreement itself is informative: it tells you that the model's computation is more complex than the attention pattern suggests. Systematic disagreements across multiple inputs might indicate that a specific head is performing a computation that the gradient-based method is missing, or that the gradient-based method is capturing a signal that attention is not routing explicitly.

Consider the architecture. Cross-attention in generation models, induction heads, and other architecturally constrained mechanisms are more interpretable via attention than feed-forward-dominated computations. When the computation you care about must pass through attention, attention is the right measurement tool.

The disagreement between attention and gradient-based methods has direct practical consequences beyond theory. We can directly measure it on a synthetic but representative example, showing how different methods rank the same tokens.

The following visualization simulates a scenario common in practice: running multiple attribution methods on the same sentence and comparing their token rankings. Methods that produce similar rankings agree on what the model finds important; methods that produce different rankings give different explanations for the same prediction. This comparison is something every practitioner should run when using interpretability tools for consequential decisions.

The correlation matrix reveals the structure of the disagreement. The three gradient-based methods (Gradient Saliency, Integrated Gradients, and Gradient-Weighted Attention) show high mutual correlations, indicating that they largely agree on which tokens are important. Raw attention shows systematically lower correlations with all three gradient-based methods, confirming that it is measuring something qualitatively different. This is not a failure of any particular method; it is evidence that attention and gradients are answering different questions about the model's computation.

Beyond the question of whether attribution methods are faithful, there is a separate question of whether they are robust: do they give consistent results across small input perturbations? An attribution method that assigns high importance to "not" for one sentence but low importance for a nearly identical sentence provides unreliable explanations, even if each individual explanation is technically faithful.

Input gradient saliency is notoriously fragile in this regard. Small, imperceptible changes to input embeddings can drastically change gradient magnitudes without changing the model's prediction. This sensitivity to irrelevant perturbations makes raw gradient saliency difficult to interpret and unstable for practical use.

SmoothGrad (Smilkov et al., 2017) addresses this by computing attributions as an average over many noisy versions of the input:

where $\epsilon_n \sim \mathcal{N}(0, \sigma^2 I)$ is Gaussian noise added to the input embedding. By averaging over many noisy inputs, SmoothGrad removes the high-frequency noise component from the gradient, producing smoother and more visually coherent saliency maps.

Integrated Gradients is inherently more robust than raw gradients because it averages along an interpolation path, which already provides a form of averaging over multiple input variants. This is one reason why Integrated Gradients is generally preferred over raw input gradients in practice.

The robustness issue applies to attention as well. While attention is generally more stable than raw gradients (softmax normalization provides some smoothing), the attention patterns for a specific layer and head can vary substantially across superficially similar inputs. Any interpretability analysis should examine consistency across multiple examples rather than drawing conclusions from single cases.

The attention-is-not-explanation debate had a significant effect on how the NLP community approaches interpretability. Before 2019, it was common to treat attention visualization as a form of explanation in publication, drawing conclusions about model behavior from heatmaps without additional justification. The work by Jain and Wallace, Wiegreffe and Pinter, and subsequent researchers raised the bar for what counts as a valid interpretability claim. Papers that rely solely on attention visualization as evidence of mechanism are now more likely to receive critical review, and interpretability conferences have developed more rigorous standards for claims about what a model is "doing."

The debate also contributed to the development of richer interpretability toolkits. The inadequacy of attention analysis motivated investment in gradient-based methods, probing classifiers, activation patching, and mechanistic interpretability, each of which provides complementary and in many cases more reliable windows into model computation. The field has moved from "look at the heatmap" to a more systematic program of studying internal representations using multiple converging methods.

The ongoing limitation is that no single interpretability method is fully satisfactory. Gradient-based methods are more faithful than attention but can be noisy and computationally expensive. Probing classifiers measure what information is present in representations but not how it is used in the final computation. Mechanistic interpretability provides the most precise analysis but requires enormous research effort per circuit identified and has so far been applied mainly to small models. Attention visualization remains the easiest and most accessible method, which means it will continue to be used, ideally with greater awareness of its limitations.

There is also a deeper epistemological issue. All interpretability methods make assumptions about what counts as an explanation, and those assumptions may not match the questions practitioners need answered. A practitioner auditing a loan decision model might need to know which applicant features drove the decision, which attention visualization cannot reliably provide and which gradient-based methods provide only in aggregate. A researcher trying to understand how a model has learned syntax might want to know about the circuit-level computations, which neither attention nor gradients directly reveal. Building interpretability methods that answer the right questions for the right use cases remains an open research problem.

Understanding what attention cannot tell you is not a reason to abandon visualization. It is a reason to use it carefully, triangulate with other methods, and resist the temptation to tell an appealing story based on a single heatmap. The history of science is full of cases where an appealing measurement was mistaken for an explanation. Attention heatmaps are persuasive precisely because they confirm our intuitions about language, but that confirmation is partly a reflection of our own prior knowledge rather than evidence about the model's internal computation.

Attention weights measure how much a query position attends to each key position, not how much that position causally influences the final prediction. The two quantities diverge whenever value vector norms are heterogeneous, when the model encodes information through residual connections, or when feed-forward layers store and retrieve information independently of attention.

The key findings from the literature are:

• Attention does not consistently correlate with gradient-based feature importance, with rank correlations often near zero for classification tasks
• Alternative attention distributions can produce identical model predictions, showing that attention is underdetermined as an explanation
• Attention can reflect syntactic structure without indicating which tokens are semantically decisive for classification
• Multi-head aggregation choices significantly affect interpretation, and no single aggregation is universally correct
• Head pruning studies show that many heads can be removed with minimal performance impact, undermining the causal interpretation of individual heads

Gradient-based alternatives, including input gradient saliency and Integrated Gradients, provide more direct measures of input sensitivity with stronger theoretical foundations. Integrated Gradients satisfies the completeness axiom, making sure that attributions sum to the model's output difference from a baseline. Hybrid methods like gradient-weighted attention combine the spatial structure of attention with gradient-based weighting to improve faithfulness while retaining interpretable spatial organization.

The appropriate role for attention analysis is exploratory: forming hypotheses to be tested with more rigorous methods, debugging obvious structural failures, and providing intuitive demonstrations of how attention operates. In architecturally constrained settings, such as induction heads and cross-attention in encoder-decoder models, attention provides more faithful signal because the computation must pass through the attention bottleneck. In general self-attention settings, treat attention patterns as one data point among many, not as the final word on what the model is computing.

The next chapter on probing classifiers provides a complementary perspective: instead of asking how information is routed (attention) or which tokens are causally important (gradients), we ask what information is present in the internal representations at each layer. Together, these three perspectives, routing, causality, and content, give a richer picture of model behavior than any one alone.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about attention analysis limitations.