Attention Analysis Limitations › The "Attention is Not Explanation" Debate

The limitations of attention as explanation were crystallized in the landmark paper by Jain and Wallace (2019): "Attention is not Explanation." Their work set off a productive but contested debate that reshaped how the NLP community thinks about interpretability. Before this paper, it was common practice to include attention heatmaps in papers as informal evidence of model behavior, with the implicit assumption that high-attention tokens were somehow responsible for the model's predictions.

Jain and Wallace examined attention-based text classification models and asked a simple question: if you change the attention weights, does the model's prediction change? Their experiments revealed two troubling findings.

The first finding: attention does not correlate with gradient-based importance. For many inputs, tokens that received high attention were not the same tokens that, when perturbed, most changed the prediction. The gradient-based measure (which does capture causal importance) and the attention-based measure often disagreed substantially. In some experiments, the Kendall rank correlation between attention and gradient importance was close to zero, indicating near-random agreement about which tokens mattered.

The second finding: alternative attention distributions can produce identical predictions. For a given input, they could find many different attention distributions that produced the same model output and the same prediction. This means attention is underdetermined: many different "explanations" (different attention patterns) are consistent with the same behavior. If attention were truly the mechanism driving the decision, such degeneracy would be impossible. You cannot claim that attention pattern A explains the prediction if attention pattern B, which looks completely different, produces the same prediction.

The practical implication is stark. If you can freely modify attention weights without changing the model's behavior, then attention weights are not reliably explaining the model's behavior. They may be a byproduct of the computation rather than the driver. A car's speedometer can be swapped for a different model without affecting the engine's operation, but we would not say the speedometer caused the car to move.

Wiegreffe and Pinter (2019) responded with "Attention is not not Explanation," arguing that Jain and Wallace's adversarial attention construction required training a separate model and therefore did not truly show that the original attention was uninformative. Their counter-argument: the existence of alternative attention distributions that produce the same prediction does not mean the original attention is not a faithful explanation, because there may be many faithful explanations.

The key distinction they drew is between the existence of alternative explanations and the faithfulness of the original. A faithful explanation accurately describes how the model computed its output. The fact that other computations could have reached the same output does not make the original computation unfaithful, any more than the existence of multiple routes to a destination means the route you took was not the one you drove.

They proposed a stricter criterion: is it possible to train a model that achieves similar performance with a completely different attention distribution? Using this criterion, they found that in many tasks, attention weights are more constrained than Jain and Wallace suggested. When the task requires routing specific information through attention (rather than through feed-forward or residual mechanisms), the attention patterns become less exchangeable.

The debate matters because it reveals that the notion of "explanation" is ambiguous. There are at least three different things you might want from an attention explanation:

• Faithfulness: Does the explanation accurately describe how the model computed its output?
• Plausibility: Does the explanation match human intuitions about which tokens are important?
• Sufficiency: Is the explanation sufficient to reproduce the model's behavior?

Attention often scores reasonably on plausibility but poorly on faithfulness and sufficiency. We naturally read attention heatmaps and think "yes, those are the relevant words," but this human plausibility does not imply the model is using them in the way we imagine. The alignment between human intuition and model attention is partly a coincidence of training data distribution, not evidence of mechanistic correspondence.

The gap between plausibility and faithfulness is worth dwelling on, because it explains why attention visualizations are so seductive and so dangerous. Humans evaluate natural language explanations using the same prior knowledge they bring to reading text. If a model attends to "bank" in the sentence "The bank was full of fish" and the model predicted a river-related sense of "bank," the attention to "fish" looks plausible because fish do swim in rivers. But the model might have reached this conclusion through the value vector for "fish" having been pushed toward river-sense representations during pretraining, with the attention to "fish" merely serving as a positional anchor rather than a disambiguating signal.

The plausibility check passes because humans and the model share background knowledge. The faithfulness check would require verifying that removing or replacing "fish" changes the prediction, which might not be the case if the river-sense representation was already established earlier in the forward pass.

This is a general pattern in interpretability research: explanations that align with human intuition are not necessarily mechanistically accurate. They may satisfy our intuitions by accident, exploiting the correlation between what we think matters and what the model has learned, without tracing the causal pathway of computation.

Subsequent work by Brunner et al. (2020) provided additional evidence against naive attention interpretability. They showed that transformers can learn to "hide" information by encoding it in residual stream representations that bypass attention. Even if you zero out certain attention connections, the model can recover information through skip connections and layer normalization.

This points to a fundamental architectural issue: attention is not the only information pathway in a transformer. The residual stream passes information directly from layer to layer. Feed-forward layers can store and retrieve factual knowledge independently of attention patterns. Attention is one mechanism among several, and measuring it alone gives an incomplete picture of information flow. The feed-forward layers in particular have been shown to function as key-value memories that store factual associations, and this storage is entirely invisible to attention analysis.

The implication for interpretability research is that any explanation method based solely on attention is necessarily incomplete. It is like trying to understand a program by only reading the function call stack and ignoring the global variables and in-place mutations. You might get a partial picture, but you will systematically miss everything that happens outside the scope of what you measured.