Attention Mechanism (2015) › Legacy and Looking Forward

Bahdanau attention replaced one fixed source encoding with a context vector that changed at every decoding step. Its use in translation also helped establish learned, content-dependent weighting as a reusable component in neural sequence models.

The transformer architecture, introduced in 2017, extended attention to relationships within a sequence. Self-attention lets each position combine information from other positions, and transformer blocks use it in place of recurrent sequence processing. Models such as GPT and BERT are built from these transformer blocks.

Attention visualizations became a common diagnostic for inspecting alignments and comparing model behavior. Later work also showed why attention weights should not be treated as a definitive explanation: other internal states and parameters contribute to each prediction.

Multimodal systems also use attention to connect representations of text, images, and audio. Cross-attention can associate a generated caption with image regions or relate a question's tokens to visual features.

The translation objective trained the alignment model jointly with the encoder and decoder. This replaced a separately engineered alignment stage with weights learned end to end, though the learned alignments were optimized for prediction rather than linguistic analysis.

Transformer language models continue to use attention, with variants that change the sparsity pattern, approximate the computation, or implement it more efficiently. Across these variants, learned weights determine how representations at different positions are combined.