mT5 › The mC4 Corpus

Building a multilingual model requires multilingual data. The mT5 team created mC4 (multilingual Colossal Clean Crawled Corpus) by applying language-specific filtering to Common Crawl web data. This process extracted text in 101 languages, resulting in a corpus orders of magnitude larger than previous multilingual datasets.

Think of Common Crawl as a photograph of the web taken repeatedly over many years, containing billions of documents in raw, unfiltered form. The mC4 pipeline takes that raw snapshot and subjects each document to a series of language detection and quality checks before keeping it. The goal is to produce a corpus where each language's subcorpus contains useful prose rather than spam, auto-generated content, or boilerplate navigation menus. The quality of the resulting corpus directly determines the quality of the representations the model can learn, which is why the construction steps deserve careful attention.

The corpus construction followed similar quality filtering steps to the English C4 we discussed in the T5 pre-training chapter, but applied language detection to separate content:

• Language identification: Each page was classified using CLD3 (Compact Language Detector), keeping only pages where the primary language exceeded a confidence threshold
• Line-level deduplication: Removed duplicate lines within each language's subcorpus to reduce boilerplate and repeated content
• Quality filtering: Applied heuristics to remove pages with too few words, excessive punctuation, or other quality issues

The resulting corpus shows extreme size variation across languages:

English contains roughly 2.7 trillion tokens, while some African languages like Yoruba have only 60 million tokens, a ratio of over 45,000:1. This imbalance creates a basic problem: training proportionally to data size would essentially ignore low-resource languages. Equal sampling would massively oversample (and overfit to) low-resource data while underutilizing high-resource content.

Notice that the size disparity has consequences beyond the number of examples. The English portion of mC4 is so large that the model would encounter many completely distinct documents during training, exposing it to enormous lexical and topical diversity. The Yoruba portion, by contrast, is small enough that the model would cycle through the same documents many times per training epoch. Cycling creates a memorization risk: the model begins to overfit specific sentences rather than learning generalizable patterns. The deeper reason why naive proportional sampling fails for low-resource languages is that repeated exposure to the same documents gives them a poor training signal.