Attention Analysis Limitations › Summary

Attention weights measure how much a query position attends to each key position, not how much that position causally influences the final prediction. The two quantities diverge whenever value vector norms are heterogeneous, when the model encodes information through residual connections, or when feed-forward layers store and retrieve information independently of attention.

The key findings from the literature are:

• Attention does not consistently correlate with gradient-based feature importance, with rank correlations often near zero for classification tasks
• Alternative attention distributions can produce identical model predictions, showing that attention is underdetermined as an explanation
• Attention can reflect syntactic structure without indicating which tokens are semantically decisive for classification
• Multi-head aggregation choices significantly affect interpretation, and no single aggregation is universally correct
• Head pruning studies show that many heads can be removed with minimal performance impact, undermining the causal interpretation of individual heads

Gradient-based alternatives, including input gradient saliency and Integrated Gradients, provide more direct measures of input sensitivity with stronger theoretical foundations. Integrated Gradients satisfies the completeness axiom, making sure that attributions sum to the model's output difference from a baseline. Hybrid methods like gradient-weighted attention combine the spatial structure of attention with gradient-based weighting to improve faithfulness while retaining interpretable spatial organization.

The appropriate role for attention analysis is exploratory: forming hypotheses to be tested with more rigorous methods, debugging obvious structural failures, and providing intuitive demonstrations of how attention operates. In architecturally constrained settings, such as induction heads and cross-attention in encoder-decoder models, attention provides more faithful signal because the computation must pass through the attention bottleneck. In general self-attention settings, treat attention patterns as one data point among many, not as the final word on what the model is computing.

The next chapter on probing classifiers provides a complementary perspective: instead of asking how information is routed (attention) or which tokens are causally important (gradients), we ask what information is present in the internal representations at each layer. Together, these three perspectives, routing, causality, and content, give a richer picture of model behavior than any one alone.