Encoder-Decoder Architecture › T5: Text-to-Text with Encoder-Decoder
Language AI HandbookAll chapters Encoder-Decoder Architecture › T5: Text-to-Text with Encoder-DecoderOne section · Sections in this chapter7 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
T5 (Text-to-Text Transfer Transformer) exemplifies the encoder-decoder paradigm at scale. Google's T5 treats every NLP task as a text-to-text problem: given some input text, generate some output text. Translation becomes "translate English to German: The cat sat" → "Die Katze saß." Summarization becomes "summarize: [long article]" → "[short summary]." Even classification becomes "sentiment positive or negative: This movie was great" → "positive."
T5 uses the standard encoder-decoder transformer architecture with relative position encodings. Both encoder and decoder share the same vocabulary and can be scaled independently. T5-Base has 220M parameters, T5-Large has 770M, T5-3B has 3 billion, and T5-11B has 11 billion parameters.
T5 Design Choices
T5 made several key architectural decisions that influenced subsequent encoder-decoder models:
-
Relative position encodings: Instead of absolute position embeddings, T5 uses relative position biases in attention, allowing better generalization to different sequence lengths.
-
Pre-norm architecture: Layer normalization applied before each sublayer rather than after, matching modern practice.
-
Shared embeddings: The same embedding matrix is used for both encoder and decoder, as well as for the output projection (tied embeddings).
-
Simplified layer norm: Removes the bias term and centering from standard layer normalization, keeping only learnable scale parameters.
T5 Model Configurations ====================================================================== Model | Layers | d_model | d_ff | Heads | Params ---------------------------------------------------------------------- T5-Small | 6 | 512 | 2048 | 8 | 60M T5-Base | 12 | 768 | 3072 | 12 | 220M T5-Large | 24 | 1024 | 4096 | 16 | 770M T5-3B | 24 | 1024 | 16384 | 32 | 3B T5-11B | 24 | 1024 | 65536 | 128 | 11B Note: Encoder and decoder each have n_layers blocks. Total layers = 2 × n_layers for encoder-decoder models.
The T5 family spans three orders of magnitude in parameter count, from 60M to 11B. The scaling pattern reveals interesting design choices: layer count increases modestly (6 → 12 → 24) while hidden dimensions and especially feed-forward dimensions grow substantially. T5-11B achieves its massive scale primarily through a 65,536-dimensional FFN, 16× larger than T5-Base. The number of attention heads also scales with model size, with T5-11B using 128 heads to maintain fine-grained attention patterns at scale.
Why Encoder-Decoder for T5?
Google explored various architectures in the T5 paper and found encoder-decoder models outperformed decoder-only models on most tasks when controlled for computational budget. The encoder's ability to process input bidirectionally proved especially valuable for:
- Tasks with long inputs: Summarization, question answering over passages
- Tasks requiring input understanding: Classification, span extraction
- Tasks with complex input-output alignment: Translation, structured generation
Decoder-only models excelled primarily at pure generation tasks without structured input. For the broad range of NLP tasks T5 targets, the encoder-decoder architecture provided better overall performance.