Attention Analysis Limitations › Visualizing Attribution Method Agreement

Michael BrenndoerferJanuary 4, 20265 min read
Language AI HandbookAll chapters
Part 59 of 67Interpretability
Chapter 448Attention Analysis Limitations › Visualizing Attribution Method Agreement
Reading section 12 of 16 · Visualizing Attribution Method Agreement · 5 minSection 12 of 16 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 448 of 513

Section 12 of 16 · 5 min read

Attention Analysis Limitations › Visualizing Attribution Method AgreementOne section · Sections in this chapter12 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The disagreement between attention and gradient-based methods has direct practical consequences beyond theory. We can directly measure it on a synthetic but representative example, showing how different methods rank the same tokens.

The following visualization simulates a scenario common in practice: running multiple attribution methods on the same sentence and comparing their token rankings. Methods that produce similar rankings agree on what the model finds important; methods that produce different rankings give different explanations for the same prediction. This comparison is something every practitioner should run when using interpretability tools for consequential decisions.

Out[8]:
Visualization
Four-column heatmap comparing token attribution scores from four interpretability methods.
Comparison of four attribution methods applied to a synthetic sentiment classification example. Raw attention (left) concentrates on structural tokens like [CLS] and common syntactic words. Gradient saliency (second column) highlights semantically decisive tokens like negations and evaluative adjectives. Integrated Gradients (third column) provides a smoother, completeness-satisfying attribution with similar semantic focus. Gradient-weighted attention (right) blends spatial structure with gradient importance. The divergence between the first column and the remaining three illustrates why attention alone is an unreliable proxy for causal importance.

The structural and semantic tokens receive dramatically different scores across the four methods. Raw attention assigns 22% of its weight to [CLS] while gradient saliency assigns only 5%. Conversely, "awful" receives 22% from gradient saliency versus only 8% from raw attention. Integrated Gradients and gradient-weighted attention fall between the extremes, with both converging on the importance of "not" and "awful" as the semantically decisive tokens. This convergence between the gradient-based methods, alongside their divergence from raw attention, supports the conclusion that attention is measuring something different from importance.

Now let's quantify the agreement between methods using pairwise rank correlations. This gives a numerical picture of how similar the different attribution methods are in their rankings of token importance.

In[9]:
Code
from scipy.stats import spearmanr

method_data = {
    "Raw Attention": raw_attn,
    "Gradient Saliency": grad_sal,
    "Integrated Gradients": ig_scores,
    "Gradient-Weighted Attn": gw_attn,
}
method_names = list(method_data.keys())
method_arrays = list(method_data.values())
n_methods = len(method_names)

# Pairwise Spearman rank correlations
corr_matrix = np.ones((n_methods, n_methods))
for i in range(n_methods):
    for j in range(i + 1, n_methods):
        rho, _ = spearmanr(method_arrays[i], method_arrays[j])
        corr_matrix[i, j] = rho
        corr_matrix[j, i] = rho
Out[10]:
Console
Pairwise Spearman Rank Correlations between Attribution Methods
-----------------------------------------------------------------
                               Raw Attention   Gradient SaliencyIntegrated GradientsGradient-Weighted Attn
           Raw Attention               1.000               0.039              -0.091               0.184
       Gradient Saliency               0.039               1.000               0.943               0.929
    Integrated Gradients              -0.091               0.943               1.000               0.890
  Gradient-Weighted Attn               0.184               0.929               0.890               1.000
Out[11]:
Visualization
Correlation matrix heatmap showing pairwise agreement between four attribution methods.
Pairwise Spearman rank correlations between four attribution methods across all tokens. High correlation (dark color) indicates methods agree on token importance rankings. The strong agreement among gradient-based methods (Gradient Saliency, Integrated Gradients, Gradient-Weighted Attention) versus their weaker agreement with Raw Attention illustrates the systematic divergence between attention-based and gradient-based attribution.

The correlation matrix reveals the structure of the disagreement. The three gradient-based methods (Gradient Saliency, Integrated Gradients, and Gradient-Weighted Attention) show high mutual correlations, indicating that they largely agree on which tokens are important. Raw attention shows systematically lower correlations with all three gradient-based methods, confirming that it is measuring something qualitatively different. This is not a failure of any particular method; it is evidence that attention and gradients are answering different questions about the model's computation.