Encoder-Decoder Architecture › The Complete Transformer Architecture
Language AI HandbookAll chapters Encoder-Decoder Architecture › The Complete Transformer ArchitectureOne section · Sections in this chapter2 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The original transformer architecture consists of three main components that work together to transform input sequences into output sequences:
-
Encoder stack: A series of encoder blocks that process the source sequence bidirectionally, producing contextualized representations for every input position.
-
Decoder stack: A series of decoder blocks that generate the target sequence autoregressively. Each decoder block contains masked self-attention for causality and cross-attention to the encoder's output.
-
Cross-attention mechanism: The bridge between encoder and decoder. At each decoder position, cross-attention queries the encoder's representations to gather relevant source information for generation.
This division of labor reflects a fundamental insight: understanding input requires different processing than generating output. The encoder sees everything at once and can use bidirectional context. The decoder must respect temporal ordering since it generates tokens sequentially, but it benefits from full access to the encoder's understanding of the source.
