Attention Mechanism (2015) › The Problem

In the 2014 sequence-to-sequence architecture, an encoder RNN processed the source sentence and passed its final hidden state to the decoder. That one fixed-size vector was the decoder's only representation of the source, creating what became known as the bottleneck problem.

For short sentences, this approach worked reasonably well. A sentence like "Hello, how are you?" contains relatively little information, and a typical hidden state size of 256 or 512 dimensions could capture the essential meaning. However, as source sentences grew longer, the fixed-size vector became increasingly inadequate. Longer sentences contained more information: additional noun phrases, modifiers, subordinate clauses, and complex grammatical structures. The encoder's final hidden state had to somehow compress all of this information while the decoder had to reconstruct the entire meaning from this single representation.

This compression problem manifested in several concrete ways. First, translation quality degraded noticeably for sentences longer than about 15 words. The model would lose track of information from earlier parts of the sentence, leading to translations that omitted important details or produced incorrect interpretations. Second, the model struggled with sentences that required maintaining long-range dependencies. In a sentence like "The keys that the man who visited yesterday left are on the table", the relationship between "keys" and "left" spans multiple words, and the final hidden state often failed to preserve this connection.

Third, the model had difficulty handling sentences with multiple independent pieces of information. A sentence like "John loves Mary, and she loves him too" contains two distinct relationships that both needed to be preserved. When compressed into a single vector, these relationships could interfere with each other or one might be lost entirely. Finally, the approach made it impossible to align specific source words with specific target words, which meant the model couldn't provide interpretable information about which source words influenced which target words.

The bottleneck problem became more severe when dealing with languages that had different word orders than the target language. In translating from English to Japanese, where the verb typically appears at the end, the encoder would process the entire English sentence before the decoder began generating Japanese. Information about the English verb, processed early by the encoder, had to be preserved through the entire encoding process and then accessed correctly by the decoder much later. The fixed-size bottleneck made this particularly challenging, as the verb information had to compete with all other sentence information for representation in the final hidden state.