T5 Pre-training › T5 Pre-training Data: The C4 Corpus

Michael BrenndoerferAugust 15, 20252 min read
Language AI HandbookAll chapters
Part 25 of 67Encoder-Decoder Models
Chapter 178T5 Pre-training › T5 Pre-training Data: The C4 Corpus
Reading section 6 of 11 · T5 Pre-training Data: The C4 Corpus · 2 minSection 6 of 11 · 2 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 178 of 513

Section 6 of 11 · 2 min read

T5 Pre-training › T5 Pre-training Data: The C4 CorpusOne section · Sections in this chapter6 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

T5 was pre-trained on the Colossal Clean Crawled Corpus (C4), a large dataset created specifically for this research. Understanding C4 helps explain T5's capabilities and limitations.

C4 Construction

C4 was derived from Common Crawl, a publicly available web scrape. The researchers applied aggressive filtering to clean the data:

  1. Language filtering: Only English text was retained, using a language detection model
  2. Quality heuristics: Pages were removed if they contained fewer than 5 sentences, had sentences shorter than 3 words, or contained offensive words from a blocklist
  3. Deduplication: Exact duplicate paragraphs were removed across the entire corpus
  4. JavaScript/boilerplate removal: Common web artifacts like "Terms of Service" and cookie notices were filtered out

The resulting dataset contains approximately 750 GB of clean text, roughly 156 billion tokens. This scale was very large at the time and enabled training much larger models than previous work.

Out[16]:
Console
C4 Corpus Statistics:
----------------------------------------
  Total size: ~750 GB
  Token count: ~156 billion
  Source: Common Crawl (April 2019)
  Languages: English only
  Filtering: Extensive quality heuristics

Quality vs. Quantity Tradeoffs

The T5 paper experimented with different levels of filtering. More aggressive filtering produced higher-quality text but reduced dataset size. The researchers found that:

  • Light filtering produced worse downstream performance despite more data
  • Heavy filtering (C4) struck a good balance between quality and quantity
  • Extremely aggressive filtering reduced the dataset too much, hurting performance

This finding influenced subsequent work on LLM pre-training, stressing that data quality often matters more than raw quantity.