T5 Architecture › Pretraining: Span Corruption

T5 uses a "span corruption" objective during pretraining, which the paper found more effective than alternatives like standard language modeling or BERT-style masked language modeling. This objective is a good challenge that forces the model to develop reliable language understanding.

The objective corrupts the input by replacing contiguous spans of tokens with single sentinel tokens, then asks the model to reconstruct those spans. This approach differs from BERT's masked language modeling, which corrupts individual tokens, and from GPT's causal language modeling, which predicts the next token given previous context. Span corruption strikes a middle ground that encourages the model to understand broader context while still learning to generate coherent multi-token sequences.

Here's the process in detail:

1. Sample span lengths from a distribution (mean length 3)
2. Select 15% of tokens total to corrupt
3. Replace each selected span with a unique sentinel token (<extra_id_0>, <extra_id_1>, etc.)
4. Create targets that contain the sentinel followed by the original tokens

For example:

• Original: The quick brown fox jumps over the lazy dog
• Corrupted input: The <extra_id_0> fox <extra_id_1> the lazy dog
• Target: <extra_id_0> quick brown <extra_id_1> jumps over

This approach forces the model to understand context deeply. It must determine what type of content belongs in each corrupted span based on surrounding words. Unlike next-token prediction (which only requires predicting one token at a time), span reconstruction requires understanding the complete context. When the model sees The <extra_id_0> fox jumps, it must recognize that the missing span should contain adjectives describing a fox, likely words like "quick brown" or "sly red." This requires understanding both syntax (adjectives precede nouns) and semantics (foxes have certain typical descriptions).

The use of contiguous spans rather than individual tokens adds another dimension of difficulty. The model cannot simply guess each missing token independently; it must generate a coherent sequence that fits grammatically and semantically as a unit. This trains the model for the kind of fluent generation required in downstream tasks like summarization and translation. Single-token masking (as in BERT) lets the model treat each masked position independently. Span masking forces the model to plan multi-token outputs, which is a closer match to the generation tasks T5 will be fine-tuned on.

The T5 paper systematically compared span corruption against several other pretraining objectives:

• Standard language modeling: predicting the next token given previous context, as in GPT
• BERT-style masked language modeling: independently masking 15% of tokens and predicting them
• Deshuffling: restoring a shuffled sentence to its original order
• Prefix language modeling: a compromise where a prefix is fed to the encoder and the suffix must be generated

Span corruption consistently outperformed these alternatives on downstream benchmarks. The key advantage over standard language modeling is that span corruption uses a bidirectional encoder, letting the model to see the full input context on both sides of each masked span. The key advantage over BERT-style masking is that span corruption requires generating multi-token outputs, which directly trains the generation capability that downstream tasks require. The key advantage over deshuffling is that span corruption preserves grammatical structure in the visible tokens, keeping the pretraining input closer to natural language.

The sentinel token approach also has an efficiency advantage. Because the target sequence contains only the masked spans (plus sentinels), it is much shorter than the full input. This reduces the computational cost of the decoder during pretraining and means the model spends most of its capacity on the interesting parts (the corrupted regions) rather than trivially copying uncorrupted text.

The sentinel tokens serve as placeholders in the input and anchors in the output, letting the model to learn which span corresponds to which sentinel. This correspondence is important. By seeing <extra_id_0> in both input and output, the model learns that whatever follows <extra_id_0> in the target is what should fill the <extra_id_0> position in the input. The sentinel approach also makes the target sequence much shorter than the original input, improving training efficiency since the model only needs to generate the corrupted portions rather than reconstructing the entire input.

T5 was trained on the Colossal Clean Crawled Corpus (C4), a 750GB dataset derived from Common Crawl. The researchers applied extensive filtering to improve quality:

• Remove pages with fewer than 5 sentences
• Discard pages containing words from a blocklist
• Remove duplicate lines across the corpus
• Keep only English text (detected by language ID)
• Remove pages with too many repetitive patterns

This cleaning produced a dataset much larger than typical pretraining corpora at the time, letting the scale experiments that T5 aimed to explore. The C4 dataset was itself a contribution of the T5 paper, and it has since been widely used as a benchmark pretraining corpus. Subsequent work on data quality, including analyses of the effect of de-duplication, quality filtering, and domain distribution, built directly on C4 as a baseline.

The scale of C4 matters for raw data volume and diversity. Common Crawl contains web pages from an enormous range of topics, writing styles, and domains. A model trained on C4 sees scientific text, news articles, forum discussions, product reviews, legal documents, and much more. This diversity is part of why T5's representations transfer well across such different downstream tasks. The breadth of the pretraining distribution means the model develops representations that generalize, rather than representations optimized for a specific domain.

One important limitation of C4 is its English-centric nature. By filtering for English text, T5's pretraining corpus excludes the majority of the world's languages. This limitation motivated the development of mT5, which we discuss below. The C4 filtering pipeline also made deliberate choices about what constitutes "quality" web text, and these choices embed assumptions about language and content that may not generalize to all use cases.