mT5 › mT5

T5's text-to-text framework demonstrated that a single model could handle diverse NLP tasks through unified formatting. However, T5 was trained exclusively on English text from the C4 corpus. This English-only focus meant the model could not process other languages without substantial additional training. mT5 (multilingual T5) addresses this limitation by extending the T5 paradigm to 101 languages while preserving the elegant text-to-text approach we covered in the T5 chapters.

The challenge of building a truly multilingual model goes beyond simply adding more training data. Languages differ dramatically in their available text resources. English dominates the web, while many languages have orders of magnitude less content. Training naively on this imbalanced data would produce a model that excels at English but performs poorly for low-resource languages. mT5 addresses this imbalance through careful corpus curation, temperature-based language sampling, and an expanded multilingual vocabulary.

To appreciate why mT5 matters, consider the languages it is designed to support. Roughly 7,000 languages are spoken in the world today, yet the overwhelming majority of NLP research has focused on fewer than a dozen of them. Even among languages with significant online presence, quality NLP tools are often limited to a handful. A speaker of Swahili, Telugu, or Amharic would find that most conversational AI systems, search engines with query understanding, and text classifiers simply do not work well in their language. The aspiration behind mT5 is that a single well-trained model could serve as the backbone for all of these use cases, letting researchers and engineers build on a common multilingual foundation rather than training separate models for every language they need to support.

The key insight motivating the entire mT5 design is that language understanding at a deep level should be largely language-agnostic. When a model learns that certain patterns of context predict certain kinds of spans, it is learning patterns in how information is structured in text, rather than properties of the English words used to express that information. This universality hypothesis drives the decision to train a single model across all 101 languages simultaneously rather than training separate models and hoping they stay compatible. The T5 chapters established the text-to-text framework and the span corruption pre-training objective. mT5 takes both of these and asks: how much can we generalize them across the world's languages while preserving the properties that made T5 effective?

Building on what we covered about T5's pre-training pipeline, the three central challenges mT5 had to solve were: where to get text in 101 languages, how to ensure all languages receive adequate training signal despite wildly different data quantities, and how to build a tokenizer that handles 101 different writing systems without exploding the model's vocabulary. Each of these challenges has a principled solution, and understanding those solutions is the subject of this chapter.