Attention Mechanism (2015) › Applications and Impact

Attention improved neural machine translation, especially on longer sentences that exposed the fixed-vector bottleneck. The decoder no longer had to recover every source detail from the encoder's final state alone.

Attention mechanisms also improved handling of long-range dependencies, which had been a persistent challenge for RNN-based models. In translating complex sentences with multiple clauses or embedded structures, attention allowed the decoder to directly access encoder states from much earlier in the sequence. A sentence like "The book that the professor who taught the advanced course recommended is excellent" contains nested dependencies spanning many words. Attention mechanisms could learn to focus on "professor" when generating the relevant target word, even if it appeared much earlier in the source sentence.

Attention matrices also provided alignment diagnostics. Researchers could inspect whether source and target positions received plausible correspondences, including cases where word order differed between languages. A diffuse or unexpected pattern could reveal a translation error, although the weights were not a complete explanation of the model's decision.

Researchers used these plots to debug alignments and compare architectures. The visualization exposed one internal weighting pattern, but did not make the rest of the network transparent.

Attention soon became a standard component of neural machine translation. Direct access to source positions helped with reordering and with copying or translating rare words and proper nouns.