Span Corruption › Limitations and Practical Considerations

Span corruption offers useful capabilities, but it comes with trade-offs that affect model behavior and downstream applications.

The most significant limitation is the mismatch between pretraining and generation tasks. During pretraining, the model learns to infill missing spans given surrounding context. During generation, the model must produce text autoregressively without such scaffolding. This gap means span-corrupted models may struggle with open-ended generation compared to models pretrained with causal language modeling. T5 addresses this partially by framing all tasks as text-to-text, but the underlying tension remains. Fine-tuning helps bridge the gap, yet models pretrained exclusively on span corruption often require more adaptation for generation-heavy applications where no clear "surrounding context" exists to anchor reconstruction.

Another consideration is span boundary artifacts. The model learns that sentinels mark span boundaries, potentially creating implicit assumptions about phrase structure. If spans happen to align with linguistic units (noun phrases, clauses), the model may learn useful structure. If spans cut through units arbitrarily, the model must learn to handle artificial boundaries. The random nature of span selection means both cases occur, which may introduce noise into the learned representations.

The reconstruction ambiguity problem is subtler but worth understanding. When the model is asked to reconstruct "quick brown" in the context "The <extra_id_0> fox jumps," the answer is unambiguous. But many natural text spans are not uniquely determined by context. "The <extra_id_0> walked into the room" could be completed with "man," "woman," "professor," "child," or hundreds of other valid continuations. The training objective penalizes all completions except the exact original text, which means the model is repeatedly penalized for generating plausible but non-original responses. This is similar to the exposure bias problem in sequence generation: the model is trained to match a specific output, but at test time there is no single correct output. Over billions of training examples, this effect averages out because the model encounters so many diverse contexts, but it means the model's generation probabilities should not be interpreted as true posterior probabilities over all valid completions.

Key practical limitations include:

• Sentinel token overhead: Each span requires a unique sentinel, consuming vocabulary space and adding tokens to both input and target sequences.
• Span length sensitivity: Very short spans (length 1) resemble MLM without the multi-prediction benefit. Very long spans remove too much context for reliable reconstruction.
• Reconstruction ambiguity: When spans contain common phrases, multiple valid reconstructions may exist, yet training penalizes all but the original.
• Pretraining-inference mismatch: At inference time on downstream tasks, no sentinel tokens appear in the input. The model must generalize from the pretraining distribution of sentinel-marked gaps to the fine-tuning distribution of task-formatted prompts.
• Dependency on encoder-decoder architecture: The full benefits of span corruption rely on having a bidirectional encoder. Applying span corruption to decoder-only models requires adaptation (such as the fill-in-the-middle variant) and loses the bidirectional encoding advantage.

Despite these limitations, span corruption works well in practice. T5 and its variants achieve strong performance across diverse benchmarks, suggesting that the benefits of efficient training and phrase-level learning outweigh the costs. The span corruption objective has also influenced subsequent work in unexpected ways: document denoising tasks, code infilling models, and even diffusion-based text generation approaches all draw on the insight that reconstruction from partial observations is a powerful self-supervised learning signal.