Span Corruption › From Token Masking to Span Corruption

Standard MLM randomly selects 15% of tokens and replaces them with [MASK]. The model predicts each masked token independently, using the surrounding context. This works well but has limitations.

To understand why, consider what exactly the model is learning. When we mask "sat" in "the cat sat on the mat," the model must predict a verb that fits syntactically and semantically in that position. This is a useful signal, and BERT showed that even individual token predictions, aggregated across billions of examples, produce powerful contextual representations. But each prediction is a local, one-shot decision. The model produces a single token and moves on.

Consider the phrase "the cat sat on the mat." If MLM masks "sat" and "mat" independently, the model makes two separate predictions. It never learns that these words might be related or that predicting multi-word expressions requires different reasoning than predicting single tokens. There is no need for the model to produce a coherent continuation across multiple positions at once.

Span corruption addresses this by masking contiguous sequences. Instead of the [MASK] sat on the [MASK], we might see the [X] on [Y] where [X] replaces "cat sat" and [Y] replaces "the mat." The model must now generate complete phrases, not isolated words. This is qualitatively different: the model must decide on the first token of a span, then condition each subsequent token on what it has already generated within that span, while staying anchored to the surrounding uncorrupted context.

Notice that this also changes the topology of the training signal. With MLM, every mask position is a separate, independent prediction. The loss is a sum of per-token cross-entropies, each depending only on the context around that individual mask. With span corruption, the target is a multi-token sequence that must be generated autoregressively. The loss for each token within a span depends on all previously generated tokens in that span, creating intra-span dependencies that MLM cannot capture at all.

This shift has several consequences. First, the input sequence becomes shorter because multiple tokens collapse into single sentinels. Second, the target sequence contains multiple tokens per sentinel, requiring the model to generate coherently. Third, the model learns about phrase-level patterns and dependencies that token-level masking might miss. Fourth, because the decoder must produce multi-token outputs for each sentinel, the model is training on a task that closely resembles real-world text generation: given context, produce a fluent, coherent continuation.

Span corruption retains MLM's overall fraction of corrupted tokens (15%) and still trains the model to fill in missing information from context. What changes is the granularity and nature of the reconstruction challenge, from isolated token prediction to phrase generation. This shift improves generation capabilities without requiring a different architecture or training infrastructure.