T5 & Text-to-Text Framework (2019) › The Solution

T5 addressed these problems by introducing a unified text-to-text framework where every NLP task was reframed as generating target text from source text. This simple but powerful insight allowed a single encoder-decoder architecture to handle all tasks without modification. The framework worked by adding task-specific prefixes to the input text, such as "translate English to German:" for translation, "summarize:" for summarization, or "cola sentence:" for the CoLA grammaticality task. The model learned to interpret these prefixes and generate the appropriate output format for each task.

The encoder-decoder architecture used in T5 was based on the original transformer design but with important refinements. Both the encoder and decoder were composed of transformer blocks with self-attention and feed-forward layers. The encoder processed the input text including the task prefix, creating representations that captured the meaning and context. The decoder then used these representations along with its own self-attention and cross-attention mechanisms to generate the target text word by word. This architecture was naturally suited for variable-length inputs and outputs, making it flexible enough to handle both understanding and generation tasks.

The training process used a novel pre-training objective called span corruption, which was designed to work well for both understanding and generation. Unlike BERT's masked language modeling that masked individual tokens, span corruption masked contiguous spans of tokens from the input text. The model was trained to reconstruct these spans in the output, learning to both understand context and generate text. This objective was more suitable for generation tasks than masked language modeling, while still providing the bidirectional understanding benefits that made BERT effective.

Span corruption worked by randomly selecting spans of tokens to mask, replacing them with sentinel tokens that the model would learn to recognize as placeholders. The input would contain the masked text with sentinel tokens, and the target would be the original spans in order, each prefixed with its corresponding sentinel token. For example, if the original text was "Thank you for inviting me to your party last week" and spans "for inviting" and "last week" were masked with sentinels <X> and <Y>, the input would be "Thank you <X> me to your party <Y>" and the target would be " for inviting <Y> last week". This taught the model to understand context and generate coherent text.

The model was pre-trained on the Colossal Clean Crawled Corpus (C4), a dataset created by filtering and cleaning Common Crawl web data. The C4 dataset contained over 700 gigabytes of high-quality text data. This provided diverse language patterns, factual knowledge, and linguistic structures. This large-scale pre-training gave T5 broad language knowledge that transferred across tasks. The pre-training process used teacher forcing, where during training the model received the correct previous tokens when generating each new token, making the learning process more stable and efficient.

After pre-training, T5 could be fine-tuned on specific tasks by providing task-specific training examples with the appropriate prefixes. The fine-tuning process was straightforward: task examples were formatted with the prefix and the model was trained to generate the target output. Unlike previous approaches that required architectural changes or specialized heads, fine-tuning T5 only involved training the model on examples in the text-to-text format. This made it easy to adapt the model to new tasks, experiment with different formulations of the same task, or combine multiple tasks in multi-task learning scenarios.

The text-to-text framework also simplified evaluation. Since all tasks produced text outputs, many tasks could be evaluated using the same metrics. Metrics like ROUGE measured overlap between generated and reference text across several generation tasks, including translation and summarization. Even classification tasks, once reframed as text generation, could be evaluated using text matching, though traditional metrics like accuracy were still used when applicable. This unification made it easier to compare performance across tasks and understand how improvements in one area might transfer to others.