Attention Visualization › Attention Flow and Residual Connections

An important subtlety in attention visualization is that attention weights describe how information is routed within a single layer's attention sub-module, but the final output of that layer also includes the residual connection. The residual connection passes the layer's input directly to the layer's output and adds it to the attention sub-module's output. This means the information at any position after a layer is a mixture of what attention routed there and what was already present before the layer.

As a consequence, even if a token places zero attention weight on another token in a given layer, the second token's information is not necessarily absent from the first token's representation after that layer. Information that was gathered by attention in earlier layers is already embedded in the representation coming into the current layer via the residual stream. Attention visualization at a single layer sees only one piece of the information routing story.

This is one reason attention rollout, which propagates information backwards through all layers while accounting for residual connections, provides a more complete picture than single-layer attention heatmaps. The residual connection also partially explains why models are robust to attention head ablation: even if you zero out an entire head's attention, the residual path continues to carry information from earlier layers.

Understanding the residual stream's role leads to a more careful interpretation of attention heatmaps. When you see a token attend weakly to distant positions in a middle layer, do not conclude that the model has not gathered that information. It may have gathered it in earlier layers and carried it forward through the residual stream. What you are seeing in the middle layer is only the incremental routing decision at that depth.