Encoder-Decoder Architecture › Summary
Language AI HandbookAll chapters Encoder-Decoder Architecture › SummaryOne section · Sections in this chapter11 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The encoder-decoder transformer architecture combines the strengths of bidirectional encoding and autoregressive decoding through the cross-attention mechanism. Key takeaways:
-
Cross-attention enables the decoder to query encoder representations, creating a bridge between source understanding and target generation. Queries come from the decoder, keys and values from the encoder.
-
Three-sublayer decoder blocks include masked self-attention for autoregressive generation, cross-attention for source access, and a feed-forward network for transformation.
-
Information flow differs between components: the encoder sees bidirectional context, the decoder's self-attention is causal, but cross-attention provides unrestricted access to the source.
-
T5 and similar models demonstrate the architecture's versatility, treating diverse NLP tasks as text-to-text problems.
-
Architecture choice depends on task structure: encoder-decoder for sequence-to-sequence transformation, decoder-only for generation, encoder-only for understanding.
The encoder-decoder paradigm from "Attention Is All You Need" remains the foundation for many production NLP systems, particularly those with structured input-output relationships. While decoder-only models dominate current scaling efforts, encoder-decoder models continue to excel at tasks that benefit from dedicated encoding and controlled generation.