Attention Intuition › The Fixed-Size Bottleneck

To understand why attention matters, we need to understand the problem it solves. In a standard encoder-decoder model, an RNN reads the input sequence one token at a time, updating its hidden state at each step. After the final token, the hidden state is passed to the decoder as the context vector.

This design places a heavy burden on a single vector. Consider translating a 20-word English sentence to French. The encoder reads all 20 words and produces a single vector. The decoder then generates the French translation, one word at a time, relying entirely on that vector.

There are two fundamental problems with this approach. First, the vector has a fixed capacity regardless of input length. A 5-word sentence and a 50-word sentence both produce context vectors of the same size. Longer inputs must compress more information into the same space, inevitably losing detail. If you picture each dimension of the context vector as a filing cabinet drawer, you have the same number of drawers whether you are filing five documents or five hundred. Something must be thrown away or badly crumpled to fit.

Second, information from early in the sequence must survive many RNN steps to reach the final hidden state. The vanishing gradient problem (explored in the Vanishing Gradients chapter) makes this difficult: gradients shrink exponentially as they propagate backward through many time steps, which means the encoder's parameters receive little signal about how to preserve early-sequence information. The context vector ends up dominated by information from the end of the input.

These two problems compound each other. Longer sequences demand more compression while simultaneously making it harder to retain early content. The empirical evidence was striking: in the years before attention, researchers consistently found that seq2seq translation quality peaked around 10 to 15 tokens and degraded sharply beyond 20 to 30 tokens. The measured decline showed that the bottleneck limited performance.

Attention addresses both problems at once. Instead of relying on a single final state, the decoder can access all encoder hidden states at every generation step. The model learns to weight these states dynamically, giving more attention to whichever encoder positions are most useful for generating the current output token.

To make the bottleneck concrete, imagine the encoder processing "The quick brown fox jumped over the lazy dog." The encoder's final hidden state must carry useful information about all nine words: the subject (fox), its modifiers (quick, brown), the verb (jumped), the preposition (over), and the object (dog) with its modifier (lazy). As the decoder generates the translation word by word, it relies entirely on this single vector for the entire sentence.

The decoder begins by generating the equivalent of "The." For this, it needs information about "The" from the encoder. Then it generates the adjective for "quick" and "brown," needing those features. By the time it reaches "dog" and "lazy," nine decoding steps later, the context vector has not changed at all; it is still the same compressed representation from the beginning. This static context vector is the bottleneck. The decoder must extract all of its needed information from one frozen vector, using it for every single generation step, regardless of where in the output it currently finds itself.

Attention shifts the problem from "compress everything into one vector" to "learn which parts are relevant right now." The latter is a much easier learning problem because it is local and specific. Rather than asking the encoder to pack every fact about a sentence into a single representation, attention lets the decoder pick up the relevant fact when it needs it.

Think of the difference between reading a book once and trying to answer questions from memory versus having the book open and being able to flip to relevant pages as you answer each question. The first approach places enormous demands on memory. The second approach makes each question easier by providing direct access to the relevant information. Attention is the second approach: the encoder states are the open book, and the attention mechanism is the ability to flip to the right page at each step.