mT5 › Limitations and Impact

mT5 represented a significant advance in multilingual NLP, but several limitations affect its practical deployment and performance. Understanding these limitations clearly helps you make good decisions about when to reach for mT5 versus a monolingual model or a more specialized multilingual architecture.

The most basic challenge is what researchers call the curse of multilinguality. As more languages are added to a fixed-capacity model, each language must share the same parameter space with all the others. The result is a capacity tradeoff: mT5-Large has roughly the same number of non-embedding parameters as T5-Large, but those parameters must represent the grammatical patterns and vocabulary of 101 languages while also encoding world knowledge. For high-resource languages like English, this dilution produces a modest but measurable drop in performance, typically 3-5 points on standard benchmarks. For applications requiring the best possible English performance, a dedicated English model like T5 or a larger mT5 variant trained with higher English weighting might be preferable. The practical message is that multilingual capability comes at a cost, and that cost grows as the number of languages increases.

Resource imbalance persists even after temperature sampling. Temperature at $T=100$ dramatically compresses the difference between English and Yoruba in sampling probability, but it cannot substitute for the underlying fact that the model sees far less total Yoruba text than English text in absolute terms. Languages with only tens of millions of tokens in mC4 develop weaker, more brittle representations. Cross-lingual transfer from English to Yoruba is accordingly less reliable than transfer to Spanish, because the Yoruba representations are less well-grounded. This perpetuates existing gaps in NLP system quality across languages, even as it narrows them. A model like mT5 is therefore not a complete solution to language inequality in AI; it is a meaningful step forward that leaves substantial gaps, particularly for the world's lowest-resource languages.

The 250K vocabulary, while much larger than T5's 32K, still cannot achieve optimal tokenization for all 101 languages simultaneously. The SentencePiece training process optimizes for the sampled training distribution, which means languages with higher sampling probability at vocabulary training time tend to get better tokenization quality. Languages where the tokenizer fragments words excessively face longer sequence lengths, higher computational costs, and effectively shorter context windows for a given maximum token budget. This tokenization efficiency gap is a concrete practical limitation when deploying mT5 in production systems where latency and cost matter.

Evaluation is also hampered by a scarcity of high-quality benchmarks for most of the languages mT5 supports. XNLI and MLQA are among the multilingual benchmarks available, alongside XTREME, but they cover at most a few dozen languages and the quality of annotations varies considerably. For the approximately 60 languages in mT5's coverage that have little or no benchmark presence, it is difficult to know how well the model performs or whether improvements to training translate into real-world benefits. This benchmark scarcity creates a measurement problem and reflects a deeper asymmetry in research investment that affects the entire multilingual NLP ecosystem.

Despite these limitations, mT5 had a major impact on the field. The model demonstrated conclusively that the text-to-text framework could scale to massive multilingual settings without basic architectural changes, validating the universality of T5's design. The mC4 corpus became a standard resource for multilingual training, used by numerous subsequent models. The zero-shot cross-lingual transfer results established new baselines that pushed the community toward more ambitious multilingual evaluation standards. For practitioners, mT5 provided an accessible off-the-shelf foundation that could be fine-tuned for multilingual classification, question answering, and generation tasks without requiring the substantial resources needed to train a multilingual model from scratch. Many production multilingual applications today, spanning translation support tools, multilingual search, and cross-lingual document classification, trace their foundations to mT5 or models directly influenced by its design.

The scaling laws we'll explore in upcoming chapters suggest that many of mT5's limitations can be addressed through increased model scale, improved training data curation, and more sophisticated sampling strategies. Larger models can support more languages with less capacity interference, better-curated corpora can improve low-resource language quality, and adaptive sampling schemes can respond dynamically to training progress rather than using a fixed temperature throughout.