T5 Architecture › Model Sizes
Language AI HandbookAll chapters T5 Architecture › Model SizesOne section · Sections in this chapter5 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
T5 was released in five sizes, letting researchers and practitioners to choose the right trade-off between capability and computational cost. Each size follows the same architecture but varies in its layer depth and width, which together determine the total parameter count.
| Model | Parameters | Layers | Hidden Size | Attention Heads | Feed-Forward Size |
|---|---|---|---|---|---|
| T5-Small | 60M | 6 | 512 | 8 | 2048 |
| T5-Base | 220M | 12 | 768 | 12 | 3072 |
| T5-Large | 770M | 24 | 1024 | 16 | 4096 |
| T5-3B | 3B | 24 | 1024 | 32 | 16384 |
| T5-11B | 11B | 24 | 1024 | 128 | 65536 |
Several patterns emerge from this scaling progression. Smaller models increase depth (more layers) as they grow, while the largest models hold depth constant and scale width instead. The feed-forward dimension grows proportionally larger at scale. The number of attention heads increases materially for T5-3B and T5-11B. This gives more specialized attention patterns.
The jump from T5-Large to T5-3B is particularly striking: the hidden dimension and number of layers stay the same, but the feed-forward dimension quadruples and the number of attention heads doubles. This asymmetric scaling shows practical findings about where additional capacity is most useful. At large scales, widening the feed-forward network (which is where factual associations tend to be stored) and adding more attention heads (which allows more diverse attention patterns) tends to give more return per parameter than simply adding more layers.
Understanding the model size trade-offs also matters for practical deployment. T5-Small (60M parameters) can run on a laptop CPU and fine-tunes in minutes on modest hardware. T5-Base strikes a balance many practitioners find useful: strong enough for production quality on many tasks, yet fast enough to iterate on. T5-Large and above typically require GPU memory measured in tens of gigabytes and training times measured in hours to days. T5-11B was primarily a research tool to study scaling behavior; for production, the smaller variants are more practical.
# Compare parameter counts across model sizes
from transformers import T5Config
sizes = ["t5-small", "t5-base", "t5-large"]
for size in sizes:
config = T5Config.from_pretrained(size)
print(f"\n{size}:")
print(f" Layers: {config.num_layers}")
print(f" Hidden size: {config.d_model}")
print(f" Attention heads: {config.num_heads}")
print(f" FF dimension: {config.d_ff}")
print(f" Vocab size: {config.vocab_size}")t5-small: Layers: 6 Hidden size: 512 Attention heads: 8 FF dimension: 2048 Vocab size: 32128
t5-base: Layers: 12 Hidden size: 768 Attention heads: 12 FF dimension: 3072 Vocab size: 32128
t5-large: Layers: 24 Hidden size: 1024 Attention heads: 16 FF dimension: 4096 Vocab size: 32128
The vocabulary size remains constant at 32,128 tokens across all sizes. This vocabulary was trained using SentencePiece on the C4 dataset, the same corpus used for pretraining. Using a fixed vocabulary across all model sizes simplifies deployment: you need only one tokenizer regardless of which T5 variant you use, and fine-tuned models can be upgraded to larger sizes without re-tokenizing training data.