Attention Intuition › Summary

Attention solves the information bottleneck in encoder-decoder models. Rather than compressing an entire input sequence into a single vector, attention allows the decoder to dynamically focus on relevant parts of the input at each generation step. The key insight is deceptively simple: instead of asking the encoder to do the impossible and preserve all information forever in a fixed vector, give the decoder the ability to look back and ask for what it needs.

Key takeaways from this chapter:

• Bottleneck motivation: Fixed-size context vectors struggle with long sequences; attention replaces the single vector with dynamic access to all encoder states
• Soft lookup: Attention functions as a differentiable dictionary lookup, computing weighted combinations of values based on query-key similarity
• Three-step process: Every attention mechanism computes alignment scores, normalizes with softmax, and produces a weighted sum of encoder states
• Interpretability: Attention weights reveal which input positions the model considers relevant, enabling visualization and debugging
• Variable-length handling: Attention scales naturally to any input length without architectural changes
• Beyond pooling: Unlike mean or max pooling, attention provides learned, context-dependent weighting that adapts to each prediction step
• End-to-end training: Attention is fully differentiable and trains jointly with the encoder and decoder, learning alignment from translation signal alone
• Decoder integration: At each decoder step, the context vector is freshly computed from a new query, enabling step-specific focus that adapts as the output evolves
• Computational trade-off: The flexibility of attention comes at $O(T^2)$ cost in sequence length for self-attention, motivating efficient variants

In the following chapters, we'll examine specific attention mechanisms in detail. Bahdanau attention introduced the additive scoring function that made attention practical for machine translation. Luong attention explored simpler alternatives including dot product and general bilinear scoring. Understanding these foundations prepares you for the self-attention mechanism at the heart of transformers, where sequences attend to themselves to build rich contextual representations without any recurrence at all.