Attention Analysis Limitations › What Attention Weights Measure

Before examining the limitations, it helps to be precise about what attention weights represent mathematically. In scaled dot-product attention, for each query token at position $i$, the model computes a distribution over all key positions $j$:

$$\alpha_{ij} = \frac{\exp(q_i \cdot k_j / \sqrt{d_k})}{\sum_{m} \exp(q_i \cdot k_m / \sqrt{d_k})}$$

where:

• $\alpha_{ij}$ is the attention weight from position $i$ to position $j$
• $q_i$ is the query vector at position $i$
• $k_j$ is the key vector at position $j$
• $d_k$ is the dimension of the key vectors, used to scale the dot products and prevent gradient vanishing in softmax

The output at position $i$ is the weighted sum of value vectors:

$$\text{output}_i = \sum_j \alpha_{ij} v_j$$

where $v_j$ is the value vector at position $j$.

The attention weight $\alpha_{ij}$ quantifies how much position $i$'s query "matches" position $j$'s key. It determines the mixing coefficients for value vectors. But note what it does not measure: it does not measure the causal effect of token $j$ on the model's final prediction. The value vectors $v_j$ are themselves transformations of the input, and different value vectors can have wildly different norms, meaning that a small attention weight to a token with a large-norm value vector might dominate a large attention weight to a token with a near-zero value vector.

This distinction between attention and importance is not merely theoretical. The final influence of a token on the model output depends on both the attention weight and the value vector. Two tokens can receive equal attention weights but have vastly different impacts if their value vectors differ in magnitude. Conversely, a token with low attention but a large-norm value vector might matter far more than the heatmap suggests.

To see why this happens, consider what value vectors represent. The key and query projections determine which tokens attend to which, but the value projection determines what information gets mixed in when attention occurs. A model trained on sentiment analysis might learn to give the value projection for sentiment-bearing tokens (words like "terrible," "excellent," "not") large norms, because these tokens carry the semantically decisive information. Meanwhile, syntactic anchor tokens like articles and prepositions might accumulate high attention weights simply because the attention mechanism uses them as positional landmarks, but their value projections carry little information about the prediction.

This is a fundamental design property of the transformer architecture. Attention weights and value vectors are independent parameters, both learned jointly. Nothing in the training objective forces a correspondence between the magnitude of attention and the magnitude of downstream effect. The model is free to learn configurations where these diverge dramatically, and in practice it often does. The model's goal is to minimize task loss, not to make attention weights interpretable.

The practical consequence is that attention-based heatmaps can be systematically misleading in exactly the settings where we most want interpretability. When a model is processing sentiment, we want to know whether it is tracking negation ("not good") versus intensifiers ("really good") correctly. Attention might show the model attending to function words for positional grounding while the actual semantic content flows through value vectors with high norms on the content words. The heatmap looks like syntactic structure; the actual computation is semantic.

There is a second architectural reason why attention weights underspecify information flow: the residual stream. In each transformer block, the output of the attention layer is added to its input through a residual connection before being passed to the feed-forward layer. This means that every token's representation at every layer contains a mix of information from the current layer's attention computation and all previous layers' outputs.

Information in the residual stream can bypass the attention mechanism entirely. A token's representation can carry task-relevant information forward from earlier layers without that information showing up in the attention weights of later layers. When you observe a late layer's attention pattern, you are seeing how that layer routes information given all the prior computation that has already been incorporated. The attention weights at layer 10 do not tell you what information was established at layer 2.

This is why zeroing out attention weights for specific tokens does not always change the model's behavior in the expected way. The model may have already encoded the necessary information in the residual stream from earlier layers, making the current layer's attention to that token redundant. Brunner et al. (2020) demonstrated exactly this: transformers can "hide" information by encoding it in residual stream representations that bypass attention, allowing the model to maintain performance even when specific attention connections are removed.