Attention Intuition › Visualizing Attention Patterns

Attention weights are typically visualized as heatmaps, with rows representing decoder steps (outputs) and columns representing encoder positions (inputs). The resulting matrix provides an intuitive view of what the model focused on when generating each output token.

Several patterns emerge from this visualization:

• Diagonal tendency: Many languages share similar word order, so attention often follows a rough diagonal path from upper-left to lower-right
• Reordering: French adjectives follow nouns ("chat noir" vs "black cat"), visible in the swapped attention for "chat" (attending to "cat") and "noir" (attending to "black")
• Many-to-one mapping: Both "etait" and "assis" attend primarily to "sat", which reflects how French uses two words where English uses one
• Article alignment: Function words like "Le" and "le" align with their English counterparts

This kind of alignment matrix became a key piece of evidence in the original Bahdanau et al. (2014) paper showing that attention had learned meaningful linguistic structure without explicit supervision. The heatmaps showed structured, interpretable correspondences that matched human linguistic intuition rather than leaving the translation process entirely opaque.