Attention Analysis Limitations › Robustness of Attribution Methods

Beyond the question of whether attribution methods are faithful, there is a separate question of whether they are robust: do they give consistent results across small input perturbations? An attribution method that assigns high importance to "not" for one sentence but low importance for a nearly identical sentence provides unreliable explanations, even if each individual explanation is technically faithful.

Input gradient saliency is notoriously fragile in this regard. Small, imperceptible changes to input embeddings can drastically change gradient magnitudes without changing the model's prediction. This sensitivity to irrelevant perturbations makes raw gradient saliency difficult to interpret and unstable for practical use.

SmoothGrad (Smilkov et al., 2017) addresses this by computing attributions as an average over many noisy versions of the input:

where $\epsilon_n \sim \mathcal{N}(0, \sigma^2 I)$ is Gaussian noise added to the input embedding. By averaging over many noisy inputs, SmoothGrad removes the high-frequency noise component from the gradient, producing smoother and more visually coherent saliency maps.

Integrated Gradients is inherently more robust than raw gradients because it averages along an interpolation path, which already provides a form of averaging over multiple input variants. This is one reason why Integrated Gradients is generally preferred over raw input gradients in practice.

The robustness issue applies to attention as well. While attention is generally more stable than raw gradients (softmax normalization provides some smoothing), the attention patterns for a specific layer and head can vary substantially across superficially similar inputs. Any interpretability analysis should examine consistency across multiple examples rather than drawing conclusions from single cases.