Span Corruption › In Practice: Using Span Corruption for Fine-Tuning

Understanding span corruption is useful for pretraining from scratch and for fine-tuning T5-based models effectively. The text-to-text framework that span corruption enables means that T5 can be adapted to almost any NLP task simply by framing the task as a text generation problem.

For classification tasks such as sentiment analysis, the input becomes "classify sentiment: [text]" and the target becomes "positive" or "negative." There is no separate classification head; the model generates the label as a text token. For extractive question answering, the input becomes "answer question: [question] context: [passage]" and the target is the answer span extracted from the passage. For abstractive summarization, the input is "summarize: [document]" and the target is the summary. In each case, the model applies exactly the same generation mechanism it learned during pretraining.

This uniformity is both a strength and a constraint. The strength is that a single model architecture and inference pipeline handles all tasks. The constraint is that task framing matters: the input prefix (the "classify sentiment:" or "summarize:" prefix) signals to the model what kind of generation is expected, and choosing the right prefix can substantially affect performance. Practitioners often experiment with several prefix phrasings and select the one that achieves the best validation performance, a process sometimes called prompt engineering even in the fine-tuning context.

Span corruption during pretraining directly trains the skills needed for this fine-tuning pattern. When the pretrained model sees <extra_id_0> in the input, it has learned: "this sentinel marks a location where I need to generate tokens." When fine-tuning reformulates tasks as "given this input, generate this output," the model applies the same generation skill. The sentinel tokens themselves are not present at fine-tuning time, but the underlying capability of context-conditioned generation transfers directly.

In practice, T5 fine-tuning proceeds as follows. You take a pretrained T5 checkpoint and a dataset formatted as input-output text pairs. You run standard seq2seq training with cross-entropy loss, updating all model weights (full fine-tuning) or just a subset (parameter-efficient methods like LoRA). Learning rates are typically much lower than pretraining (1e-4 to 1e-3 range), and training runs for far fewer steps. The model converges quickly because the pretrained representations already encode strong linguistic knowledge; fine-tuning only needs to specialize this knowledge for the specific task distribution.