mT5 › Summary

mT5 extends the T5 text-to-text paradigm to 101 languages through several interconnected innovations that address the unique challenges of multilingual pre-training:

• mC4 corpus: A massive multilingual dataset extracted from Common Crawl, with language-specific filtering and CLD3 language detection applied to create quality-filtered subcorpora for each supported language
• Temperature-based sampling: Uses $p_l^{(T)} = |D_l|^{1/T} / \sum_{l'} |D_{l'}|^{1/T}$ with $T=100$ to balance between proportional and uniform language sampling, boosting low-resource languages by orders of magnitude relative to their raw corpus sizes
• Expanded vocabulary: 250K SentencePiece tokens (vs. 32K for T5) to cover diverse scripts and morphological patterns, trained with the same temperature sampling to ensure adequate vocabulary coverage for low-resource languages
• Cross-lingual transfer: Learns language-agnostic representations through span corruption across all 101 languages simultaneously, letting fine-tuning on English data and reasonable evaluation performance on other languages
• Performance tradeoffs: Slightly lower English performance compared to T5 due to the curse of multilinguality, but gains 100 additional languages with strong multilingual and cross-lingual transfer capabilities

The central design philosophy of mT5 is that the challenges of multilingual learning are engineering problems that can be addressed with principled solutions, rather than basic barriers requiring architectural changes. Temperature sampling handles resource imbalance. Vocabulary expansion handles script diversity. The span corruption objective, unchanged from T5, handles representation learning across all languages simultaneously. This philosophy has proven highly influential: the temperature sampling formula and multilingual tokenization strategies pioneered by mT5 appear in nearly every subsequent large-scale multilingual model, from BLOOM to mBART to modern multilingual instruction-tuned systems. The core insight that a single encoder-decoder model can serve as a multilingual foundation for dozens of downstream tasks remains one of the most practically significant contributions of the mT5 paper.