Attention Intuition › Attention Weight Interpretation

Attention weights offer something rare in deep learning: interpretability. By examining which input positions receive high attention weights, we can understand what the model is "looking at" when making each prediction.

In machine translation, attention weights often reveal meaningful alignments between source and target languages. When translating "The black cat sat on the mat" to French "Le chat noir etait assis sur le tapis", the model typically attends to "cat" when generating "chat", "black" when generating "noir", and "mat" when generating "tapis." This alignment emerges automatically from training, without any explicit supervision about word correspondences.

The alignment patterns that emerge often match what a linguist would predict. Subject-verb-object structures tend to produce diagonal attention matrices for languages with similar word order. Cross-linguistic reorderings, such as the adjective-after-noun convention in French, produce off-diagonal patterns where target tokens attend to source positions in a different order. The model discovers these patterns without being told anything about grammar or syntax; it learns them because they help minimize translation loss.

However, attention weights require careful interpretation. Several caveats apply:

• Weights show correlation, not causation: High attention on a position doesn't mean that position caused the output. It means the model found that position useful, but the relationship may be indirect. A model translating "not happy" might attend heavily to "not" while generating the translation, but the actual semantic work of understanding negation is distributed across many layers and weights in the network.
• Distributed information: Important information might be spread across multiple positions with moderate weights. Focusing only on the highest-weight position misses the full picture. Some linguistic relationships, such as long-distance agreement, require integrating information from multiple positions simultaneously.
• Layer effects: In multi-layer models, different layers may attend to different aspects of the input. Shallow layers often capture local structure while deeper layers capture longer-range dependencies.
• Head specialization: In multi-head attention, different heads often specialize in different types of relationships. One head might track positional adjacency, another might track syntactic dependencies, and another might track semantic similarity. Visualizing only one head can give a misleading picture of what the model has learned.

Despite these caveats, attention visualization remains one of the most valuable tools for understanding model behavior and debugging unexpected predictions. When a translation is wrong, plotting the attention heatmap often reveals immediately whether the model attended to the wrong source position, skipped a word, or duplicated coverage. This transparency helped practitioners diagnose errors and contributed to attention's rapid adoption.

Attention heatmaps place source tokens on one axis and target tokens on the other, with cell intensity representing the attention weight. A few patterns are worth knowing:

A near-diagonal pattern indicates that the source and target languages have similar word order, and the model is generating the translation in roughly the same order as the source. This is common for closely related languages like English and German or Spanish and Italian.

Vertical stripes indicate that the same source position receives high attention across many target steps. This often corresponds to source tokens that influence many target words, such as articles, auxiliary verbs, or punctuation that triggers systematic differences between languages.

Horizontal stripes indicate that a specific target token attends broadly across many source positions. This can occur when generating punctuation, discourse markers, or words whose meaning depends on the global context rather than a specific source word.