mT5 › mT5 vs T5 Performance
Language AI HandbookAll chapters mT5 › mT5 vs T5 PerformanceOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Comparing mT5 and T5 reveals the tradeoffs involved in multilingual training. On English-only benchmarks, mT5 slightly underperforms T5. This reflects the "curse of multilinguality": the model must divide its capacity across many languages.
The benchmark comparison reveals consistent but modest performance gaps: mT5-Large scores approximately 3 points lower on GLUE (86.4 vs 89.7), SuperGLUE (80.9 vs 84.6), and both SQuAD variants. The ROUGE-L gap on CNN/DM is about 2.8 points. This English performance gap is relatively small, typically 3-4 points, while mT5 gains the ability to process 100 additional languages. For applications requiring multilingual support, this tradeoff is highly favorable.
Notice that the performance gap narrows with model scale. At the XXL level, the sheer number of parameters provides enough capacity that the curse of multilinguality becomes less severe. This is an important empirical observation: the English penalty you pay for multilingual training is not a fixed tax, it is a function of model size relative to the number of languages. With enough parameters, a single multilingual model can come very close to monolingual performance while still supporting dozens of additional languages. This finding has important practical implications for system design, particularly as very large models become more accessible for fine-tuning through parameter-efficient methods like LoRA and prefix tuning, which we will explore in later chapters.
Performance Across Model Sizes
mT5 was released in multiple sizes, following T5's scaling approach:
# mT5 model variant specifications
# Format: (name, parameters, layers, d_model, d_ff)
variants = [
("Small", 300_000_000, 8, 512, 1024),
("Base", 580_000_000, 12, 768, 2048),
("Large", 1_200_000_000, 24, 1024, 2816),
("XL", 3_700_000_000, 24, 2048, 5120),
("XXL", 13_000_000_000, 24, 4096, 10240),
]
# Calculate total embedding parameters for largest variant
xxl_vocab_params = 250_000 * variants[-1][3] # vocab_size * d_modelmT5 Model Variants: Variant Parameters Layers d_model d_ff -------------------------------------------------------- Small 300,000,000 8 512 1024 Base 580,000,000 12 768 2048 Large 1,200,000,000 24 1024 2816 XL 3,700,000,000 24 2048 5120 XXL 13,000,000,000 24 4096 10240
The table shows how mT5 scales from 300M to 13B parameters. Each size increase brings proportionally larger hidden dimensions (d_model) and feed-forward dimensions (d_ff), with the largest models using 24 layers consistently. Larger variants show better cross-lingual transfer, suggesting that additional capacity helps the model maintain stronger representations across more languages. The relationship between model size and multilingual performance is an area we'll explore further in the scaling laws chapters.
The parameter counts show that the embedding layer grows faster than the total parameter count as d_model increases. For mT5-Small, the 250K vocabulary with 512-dimensional embeddings produces 128M embedding parameters, roughly 43% of the total 300M. For mT5-XXL with 4096-dimensional embeddings, the vocabulary accounts for 1.024B parameters out of 13B total, a smaller fraction but still a substantial absolute count. This explains why efficient embedding representations, such as tied input and output embeddings (which mT5 uses, following T5), matter especially in multilingual settings where the vocabulary is large.