Attention Mechanism (2015) › 2015: Attention Mechanism

In work published in 2015, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced an attention mechanism for neural machine translation. It allowed the decoder to compute a different weighted combination of source representations for each generated word. The method addressed a limitation of early encoder-decoder systems, which compressed the source sentence into one fixed-size vector.

Statistical machine translation systems used pipelines with separate stages for phrase extraction, translation, and reordering. Sequence-to-sequence architectures offered an alternative in 2014: an encoder and decoder could be trained together to map source text to target text. Their reliance on a single fixed-size source representation, however, created an information bottleneck.

Sequence-to-sequence models worked by having an encoder RNN process the source sentence and compress it into a single fixed-size vector, which a decoder RNN then used to generate the target sentence. While this approach showed promise for short sentences, it struggled with longer sequences. The encoder's final hidden state had to capture all information about the source sentence in a single vector, creating an information bottleneck that made it difficult to handle sentences longer than about 15 words. Longer source sentences led to degraded translation quality as important details were lost in the compression process.

Bahdanau and his colleagues removed that constraint by giving the decoder access to every encoder hidden state. When translating "the cat sat on the mat" to French, the model could assign high weight to "cat" while generating "chat," then change the weights for "s'est assis" or "tapis." The learned weights improved translation and produced useful source-target alignment visualizations.

For a detailed derivation and implementation of the additive alignment model, continue with Bahdanau Attention: Dynamic Context for NMT in the Language AI Handbook.