Decoder Architecture › The Decoder-Only Design
Language AI HandbookAll chapters Decoder Architecture › The Decoder-Only DesignOne section · Sections in this chapter2 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The original transformer from "Attention Is All You Need" (2017) used both an encoder and a decoder for machine translation. For language modeling, the encoder is not required. A stack of decoder blocks, trained to predict the next token, learns rich representations of language without any encoder component.
A decoder-only model consists of a stack of transformer blocks that process tokens autoregressively. Each block uses causal (masked) self-attention to ensure that predictions at position depend only on positions .
The decoder-only approach offers several advantages:
- Simplicity: One stack of layers instead of two (no encoder-decoder attention needed)
- Unified architecture: The same model handles both understanding and generation
- Scalability: Easier to scale parameters when you have a single stack
- Pretraining efficiency: Next-token prediction provides dense supervision at every position
GPT-1, GPT-2, GPT-3, and their successors all use this decoder-only design. Many open-source models, including Llama and Mistral, use it as well. The architecture has become the default choice for generative language AI.