T5 Architecture › The Text-to-Text Framework

T5's central insight is that natural language gives a universal interface for NLP tasks. Instead of building task-specific architectures with specialized output heads, T5 learns to generate the answer as text. This means the same model, loss function, and training procedure work for any task.

To appreciate why this matters, consider how NLP systems were traditionally built. Classification tasks required a final layer that mapped hidden representations to a fixed set of class probabilities. Question answering systems needed span prediction heads that identified start and end positions in text. Translation models required sequence-to-sequence architectures with dedicated vocabulary handling for each language pair. Each task demanded its own architectural modifications, training objectives, and output processing logic. When practitioners wanted to build a system that handled multiple tasks, they faced a choice: build and maintain several separate models, or construct multi-task architectures with complex shared backbones and task-specific heads. Neither option was particularly elegant.

The text-to-text framework eliminates these distinctions by building on an insight: natural language is already a universal representation system. Humans express classification decisions ("this is negative"), answer questions ("the color is blue"), and produce translations ("Das Haus ist wunderbar") all through the same medium, text. If we train a model to be exceptionally good at generating text, it can express any answer we might need. The format of the output encodes the task semantics, not the architecture. A model that generates "positive" when asked about sentiment is performing classification just as validly as one that outputs a probability vector.

Consider how different tasks map to this framework:

• Translation: translate English to German: The house is wonderful. → Das Haus ist wunderbar.
• Summarization: summarize: [long article text] → [concise summary]
• Classification: sentiment: This movie was terrible. → negative
• Question answering: question: What color is the sky? context: The sky appears blue during the day. → blue

The task prefix tells the model what to do, and the target text encodes the answer. Classification becomes generating the class label as a word. Regression could output the number as text. Even complex structured outputs like parse trees can be serialized as strings. The key constraint is that the model must learn to associate each prefix with the correct behavior during pretraining and fine-tuning. This is less fragile than it might sound: natural language is rich enough that even short prefixes like "summarize:" or "translate English to German:" carry strong signals about what kind of output is expected.

This uniformity has practical benefits. You can fine-tune a single model on multiple tasks simultaneously. You can add new tasks without changing the architecture. You can also use text generation advances, like beam search and nucleus sampling, across all applications. Perhaps most importantly, unified training allows tasks to share representations. A model that learns to translate English to French and English to German does not learn two unrelated skill sets. It learns representations of English meaning that transfer across both translation directions, and this sharing often improves performance on both tasks.

The text-to-text framing also has a subtle pedagogical advantage: it makes the model's "reasoning" more transparent. When a sentiment classifier outputs a probability distribution over classes, it is hard to interrogate what the model understood. When T5 outputs "negative" or "positive" as text, you can probe why it chose that word, ask follow-up questions, or even request explanations. This interpretability benefit becomes more significant in the era of larger models and chain-of-thought prompting, where the output text itself is a reasoning trace.

The T5 paper used simple, human-readable prefixes for each task. For the GLUE benchmark tasks, prefixes like "sst2 sentence:" (for sentiment) and "cola sentence:" (for grammatical acceptability) were used. These prefixes were not learned automatically; they were hand-designed to be descriptive of the task. The model's ability to generalize to new tasks at inference time depends on the prefix resembling the task descriptions it saw during training. This is why instruction-tuned successors like Flan-T5 invest heavily in diverse and natural-sounding instruction formats: if you train on hundreds of different phrasings of the same task, the model becomes more reliable to novel instructions that differ from any specific training example.

Notice that this framing requires the vocabulary to include the class labels as tokens. For sentiment classification, "positive" and "negative" must be representable tokens, which they are in T5's SentencePiece vocabulary. For tasks with unusual label spaces (like predicting numerical ratings or specialized codes), you may need to ensure your labels are covered by the tokenizer. This is rarely a problem in practice because T5's 32,128-token vocabulary covers most English words and common numeric representations, but it is worth keeping in mind when adapting the framework to highly specialized domains.