T5 Task Formatting › Summary

T5's text-to-text paradigm turns NLP by unifying all tasks into a single sequence-to-sequence generation framework. Every task becomes a text transformation: given an input string, produce an output string. Task prefixes signal which operation to perform, while consistent formatting lets multitask training and positive transfer between tasks. Classification becomes generating label text, NER becomes listing or marking entities, QA becomes generating answers conditioned on a question and optional context, and structured outputs like JSON or SQL become text generation targets.

The key insights from this chapter:

• Task prefixes act as routing instructions, conditioning the model on which transformation to apply. Natural language prefixes work better than arbitrary codes because they activate the model's existing linguistic knowledge.
• Classification as generation works by having the model output label text, with prediction probabilities recoverable from decoder logits by scoring the likelihood of each candidate label.
• NER as generation can use either entity listing or inline markup formats, each with different tradeoffs between parsability, positional information, and generation complexity.
• QA naturally fits text-to-text, letting both extractive and abstractive answers, as well as both open-book QA with provided context and closed-book QA that draws on parametric memory from pre-training.
• Multitask training becomes trivial since all tasks share the same input/output format, and mixing diverse tasks lets positive transfer that benefits all tasks simultaneously.
• Output parsing is needed in production systems because generated text may vary from expected formats, and good parsers handle common variations gracefully while failing informatively on truly malformed outputs.
• Limitations include generation overhead for simple tasks, output format inconsistency, error accumulation in structured outputs, and vocabulary constraints for uncommon labels.

The next chapter introduces BART, another encoder-decoder model that takes a different approach to pre-training by corrupting text in diverse ways and learning to reconstruct the original, while maintaining similar flexibility for downstream text generation and understanding tasks.