Span Corruption › Summary

Span corruption extends masked language modeling by corrupting contiguous token spans rather than individual tokens. Key takeaways:

• Span selection: Geometric distributions with mean 3 balance short and long spans, exposing the model to both token-level and phrase-level reconstruction.
• Sentinel tokens: Unique sentinels (<extra_id_0>, <extra_id_1>, ...) replace each span, creating a clear mapping between input positions and target outputs.
• Sequence-to-sequence framing: The corrupted input becomes the encoder source; concatenated spans with sentinels become the decoder target.
• Computational efficiency: Shorter input and target sequences reduce attention costs, making training more efficient than full-sequence objectives.
• Trade-offs: The infilling objective may not transfer perfectly to open-ended generation, requiring careful fine-tuning for generation tasks.

Span corruption powers T5 and influenced subsequent models like UL2 and PaLM, which explore mixtures of denoising objectives. Understanding this technique provides insight into how pretraining objectives shape model capabilities and efficiency. The broader lesson is that self-supervised pretraining objectives are not arbitrary: the structure of the pretraining task directly shapes what the model learns to do, and choosing an objective that mirrors the structure of downstream tasks is a key lever for building capable, transferable models.