T5 Pre-training › Summary

T5 pre-training centers on span corruption, a denoising objective that teaches the model to reconstruct corrupted portions of text. The key innovations include:

• Span-level masking: Rather than individual tokens, T5 masks contiguous spans of text with an average length of 3 tokens. This gives richer training signal than single-token prediction.

• Sentinel tokens: Each corrupted span is replaced by a unique sentinel token (<extra_id_0>, <extra_id_1>, etc.), creating a compact input representation. The model learns to map these placeholders back to their original content.

• Efficient targets: The target sequence contains only the corrupted spans, not the full original text. This makes training more computationally efficient while still requiring complete understanding of context.

• The C4 corpus: T5 introduced a carefully filtered 750 GB dataset derived from Common Crawl. This shows that data quality materially impacts downstream performance.

• Scale studies: With models ranging from 60 million to 11 billion parameters, T5 established clear scaling relationships and provided the community with models at various resource requirements.

The span corruption objective, combined with the text-to-text framework, produced models that achieved state-of-the-art results across diverse benchmarks. While subsequent decoder-only models have dominated headlines, T5's encoder-decoder architecture and pre-training approach remain highly effective for tasks with clear input-output structure, continuing to influence modern NLP systems.