Attention Analysis Limitations › Worked Example: Comparing Attention vs. Gradient Importance

To make the divergence concrete, let's walk through a small example. Suppose a sentiment classifier processes the sentence: "The movie was not terrible, it was genuinely awful."

If we extract the attention weights from a specific head at a specific layer, we might see high attention to "movie" and "was" from the [CLS] token, because these are common syntactic anchors. The words "terrible" and "awful" might receive moderate attention. Syntactic anchors attract attention not because they are semantically decisive but because they serve as positional landmarks that help the model orient itself within the sentence structure. Function words and common nouns are often the most predictable tokens in context, making them useful attention targets for structural grounding.

But if we compute input gradients, "not" and "awful" typically receive the highest scores, because flipping the sentiment requires changing these specific tokens. The word "was" is irrelevant to sentiment: you could replace it with "seemed" and the prediction stays identical. The gradient correctly identifies "not" as high-importance because negation is what transforms the sentence from a negative-without-hedge ("the movie was terrible") to a double-negation with different valence ("it was not terrible"). The gradient for "awful" is high because it is the strongest sentiment word in the sentence, whose replacement with a neutral noun would dramatically change the prediction.

This is the practical failure mode. A researcher looking only at attention might conclude "the model is attending to 'movie' and 'was,'" which sounds like syntactic structure learning. A researcher looking at gradients concludes "the model is sensitive to 'not' and 'awful,'" which correctly identifies the semantically relevant negation and intensifier. These are different stories, and the gradient story is the more faithful one. The attention story is not wrong per se (the model does attend to those tokens), but it answers a different question than the one the researcher usually has in mind.

The same divergence appears in question answering tasks, where attention frequently distributes across the question and context tokens for structural grounding, while gradients identify the specific evidence tokens whose content determines the answer. In machine translation, attention often aligns with grammatical boundaries while gradients show the content words that most constrain the target translation. The pattern is consistent: attention captures routing structure; gradients capture semantic influence.