Span Corruption › Span Corruption
Language AI HandbookAll chapters Span Corruption › Span CorruptionOne section · Sections in this chapter1 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Masked language modeling (MLM) trains models to predict individual tokens hidden behind [MASK] placeholders. But what if we masked entire phrases or multi-word expressions instead? Span corruption takes this idea further: rather than masking isolated tokens, it corrupts contiguous spans of text and asks the model to reconstruct them. This simple shift fundamentally changes what the model learns during pretraining.
T5 (Text-to-Text Transfer Transformer) popularized span corruption as its primary pretraining objective. By treating the corrupted input as source text and the original spans as target text, T5 frames pretraining as a sequence-to-sequence problem. This approach produces models that excel at generation tasks while remaining competitive on understanding benchmarks.
Think of the difference between patching a handful of isolated holes in a wall versus repairing entire sections that have been stripped away. Patching individual holes is a targeted, local task. Restoring whole sections requires understanding the structure of the wall, the relationship between surrounding elements, and how to produce a coherent surface. Span corruption gives the model the harder, more structurally demanding task, which builds richer internal representations.
The key insight behind span corruption is that natural language is not a bag of independent tokens. Words cluster into phrases, phrases into clauses, and clauses into sentences. Noun phrases, verb phrases, and named entities are coherent linguistic units whose meaning is distributed across multiple tokens. By masking and reconstructing these multi-token units, the model is forced to internalize phrase-level structure and produce text that is locally coherent, not just statistically plausible token by token.
In practice, span corruption also produces computational benefits that make it attractive for large-scale pretraining. Because each corrupted span collapses into a single sentinel token, the input sequence shrinks. Attention computation, which scales quadratically with sequence length, becomes cheaper. The model sees nearly the same amount of text but processes it in a more compact, information-dense form. These savings compound at the scale of hundreds of billions of pretraining tokens.
In this chapter, you'll learn how span corruption works, why it outperforms token-level masking for certain applications, and how to implement it from scratch. We'll cover span selection strategies, the mathematics of span length distributions, sentinel token design, and the computational advantages that make this approach attractive at scale.
T5 was introduced by Raffel et al. at Google in 2019 in the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." The central thesis was that every NLP task, whether classification, translation, summarization, or question answering, could be cast as a sequence-to-sequence problem where both inputs and outputs are text strings. This unification allowed a single model architecture and training procedure to handle an enormous variety of tasks without task-specific heads or loss functions. Span corruption was the pretraining objective that made this text-to-text framing work at scale: by treating corrupted spans as the source and original spans as the target, pretraining itself became a sequence-to-sequence task. T5 was pretrained on the Colossal Clean Crawled Corpus (C4), a 750 GB filtered web text corpus, and the resulting models became foundational to a generation of encoder-decoder architectures.