T5 Task Formatting › Multitask Training

A key consequence of unified task formatting is the ability to train on multiple tasks simultaneously. This forms a different training paradigm: learning one task can improve performance on others through positive transfer. When all tasks use the same input-output format, they can be mixed freely in the same training batch without any changes to the training loop.

Think of multitask training as cross-training for a language model. A runner who only trains on flat ground may be fast on flat courses but struggle on hills. A runner who trains on both hills and flat surfaces develops leg strength and cardiovascular fitness that transfers between environments. Similarly, a model that trains only on sentiment classification learns to identify opinion words and sentence-level sentiment polarity. A model that also trains on translation develops a deeper understanding of semantic equivalence across languages, which benefits its sentiment understanding. A model that trains on QA develops reasoning skills that benefit its comprehension across all tasks.

T5's original training mixed examples from many tasks, with task proportions tuned to balance learning across all tasks. This multitask setup lets the model to develop reliable representations that transfer across tasks. Skills learned from translation help summarization because both require understanding semantic equivalence. Reasoning developed through QA improves NLI performance because both require logical inference. The shared encoder learns representations that capture linguistic information useful across all tasks, while the decoder learns flexible generation strategies that adapt to different output formats based on the prefix.

The intuition behind why multitask learning helps is that different tasks emphasize different aspects of language understanding. Translation forces the model to deeply understand semantics in order to preserve meaning across languages. Summarization teaches the model to identify salient information and compress it without losing key facts. QA develops fact retrieval and reasoning. Classification hones sensitivity to document-level properties such as sentiment and topic, including stylistic properties. When these tasks share parameters, the model must develop representations that serve all these needs simultaneously. Such representations tend to be richer and more generalizable than those learned for any single task.

Task proportions in multitask training are not arbitrary. T5's authors experimented with different mixing strategies and found that equal mixing does not necessarily yield the best results. Tasks with large datasets, like translation, can dominate if examples are mixed uniformly, potentially causing the model to over-optimize for those tasks at the expense of smaller tasks. Strategies like temperature-based mixing, where task sampling probability is adjusted by a temperature parameter, allow practitioners to balance learning across tasks of different sizes. This is an important hyperparameter in T5-style multitask training that deserves careful tuning for any custom application.