Attention Analysis Limitations › Code Implementation
Language AI HandbookAll chapters Attention Analysis Limitations › Code ImplementationOne section · Sections in this chapter6 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We implement a self-contained demonstration using NumPy to simulate the core divergence between attention weights and gradient-based importance. The key phenomenon, that attention routing does not predict gradient sensitivity, emerges from a concrete property: different tokens have value vectors of different norms, so a small attention weight multiplied by a large-norm value vector can contribute more to the output than a large attention weight multiplied by a small-norm value vector.
We set up the token sequence and define simulated attention weights and value vector norms to mirror what research has found in real transformer models.
import numpy as np
# Tokens for the sentence: "The film was not terrible ; it was genuinely awful ."
tokens = [
"[CLS]",
"the",
"film",
"was",
"not",
"terrible",
";",
"it",
"was",
"genuinely",
"awful",
".",
"[SEP]",
]
n_tokens = len(tokens)
# Attention weights: concentration on structural tokens ([CLS], "the", "film", [SEP])
# This mirrors the well-documented pattern in transformer language models
raw_attn_weights = np.array(
[
0.22,
0.09,
0.10,
0.09,
0.06,
0.07,
0.04,
0.05,
0.07,
0.06,
0.08,
0.05,
0.12,
]
)
raw_attn_weights /= raw_attn_weights.sum() # Normalize to sum to 1Next we define value vector norms per token. The key design choice mirrors how transformers work: sentiment-bearing tokens ("not", "terrible", "genuinely", "awful") develop large-norm value vectors through training, because their semantic content is critical for downstream predictions. The structural tokens attend heavily but have smaller value vector norms, because their role is positional orientation rather than semantic content injection.
# Value vector norms per token: semantic tokens have larger norms
# Structural tokens have small norms despite high attention weight
value_norms = np.array(
[
0.30,
0.40,
0.50,
0.40,
2.10,
1.80,
0.20,
0.40,
0.40,
1.70,
2.30,
0.30,
0.30,
]
)
# True influence = attention_weight * value_norm * downstream_projection
# Simplified: influence_i proportional to attn_i * value_norm_i
# (In full transformers, the downstream projection adds another scalar factor)
downstream_proj = np.abs(np.random.randn(n_tokens)) * 0.5 + 0.5
influence = raw_attn_weights * value_norms * downstream_proj
# Gradient saliency approximation: proportional to influence
# (This is exact for a linear model; for nonlinear models it's a first-order approximation)
gradient_saliency = influence.copy()We normalize both attention and gradient scores to [0, 1] for comparison, then compute Spearman rank correlation to quantify how much the two methods agree on the relative importance of tokens.
def normalize_scores(scores):
mn, mx = scores.min(), scores.max()
return (scores - mn) / (mx - mn + 1e-8)
attn_norm = normalize_scores(raw_attn_weights)
grad_norm = normalize_scores(gradient_saliency)
# Compute Spearman rank correlation to quantify divergence
attn_ranks = np.argsort(np.argsort(-attn_norm)) # Rank from highest to lowest
grad_ranks = np.argsort(np.argsort(-grad_norm))
rank_diff = attn_ranks - grad_ranks
spearman_rho = 1 - 6 * np.sum(rank_diff**2) / (n_tokens * (n_tokens**2 - 1))Spearman rank correlation (attention vs. gradient): 0.341 Token Attn weight Value norm Grad saliency ------------------------------------------------------- [CLS] 1.000 0.048 0.314 the 0.278 0.095 0.082 film 0.333 0.143 0.248 was 0.278 0.095 0.282 not 0.111 0.905 0.542 terrible 0.167 0.762 0.542 ; 0.000 0.000 0.000 it 0.056 0.095 0.059 was 0.167 0.095 0.082 genuinely 0.111 0.714 0.550 awful 0.222 1.000 1.000 . 0.056 0.048 0.005 [SEP] 0.444 0.048 0.097
The Spearman rank correlation is substantially below 1.0, confirming that attention weights and gradient-based importance give different rankings. Notice that [CLS] and structural tokens have the highest attention but modest gradient saliency, while "not," "terrible," "genuinely," and "awful" have relatively low attention but high gradient saliency because their large value vector norms amplify their downstream influence.
Now let's visualize this divergence as a comparison heatmap.
