Attention Intuition › Learned Attention Patterns
Language AI HandbookAll chapters Attention Intuition › Learned Attention PatternsOne section · Sections in this chapter13 / 20 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
What would attention look like in a trained model? The table below shows realistic attention patterns for four sentiment sentences:
| Sentence | the | movie/acting/plot | was | sentiment word | other |
|---|---|---|---|---|---|
| "the movie was great" | 0.05 | 0.15 | 0.10 | 0.70 (great) | . |
| "the movie was terrible" | 0.05 | 0.12 | 0.08 | 0.75 (terrible) | . |
| "the acting was not great" | 0.03 | 0.12 | 0.05 | 0.35 (great) | 0.45 (not) |
| "really loved the plot" | 0.05 | 0.25 (plot) | . | 0.55 (loved) | 0.15 (really) |
These patterns reveal what we would expect:
- Sentiment words dominate: "great", "terrible", and "loved" receive the highest weights
- Negation matters: In "not great", both "not" and "great" receive significant attention, as the model must combine them to understand the negated sentiment
- Function words ignored: Words like "the" and "was" consistently receive low attention
The negation case deserves special emphasis. Negation is notoriously difficult for bag-of-words models because the word "great" has a positive sentiment regardless of context. A model that attends to "not" alongside "great" has implicitly learned that negation modifies the adjacent sentiment word, even though the model was only trained on sentiment labels, not on any explicit representation of negation. This is attention learning linguistic structure from raw data, which is exactly the kind of emergent behavior that made researchers excited about it in 2014 and remains impressive today.