T5 Architecture › T5 Architecture

The Text-to-Text Transfer Transformer (T5) introduced a unifying framework for natural language processing: treat every task as text generation. Translation, summarization, question answering, and classification all become the same problem of mapping input text to output text. This simple approach allowed researchers to study what matters most for transfer learning at scale, and the answers they found reshaped how the field thinks about model design, pretraining objectives, and the relationship between architecture and capability.

Released by Google Research in 2019, T5 emerged from a systematic exploration of pre-training techniques, model architectures, and scaling strategies. The accompanying paper, "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," tested dozens of design choices to identify which ones mattered. The result was both a powerful model family and a practical guide to building better language models. Rather than proposing a single novel trick and claiming it as the source of improved performance, the T5 researchers ran a controlled ablation study at a scale that few teams could match, making their findings unusually trustworthy.

T5's encoder-decoder architecture handles both understanding and generation in a single model. The encoder processes the full input with bidirectional attention, while the decoder generates output autoregressively. This design excels at tasks requiring deep comprehension of the input before creating structured output. Translation and summarization fit this pattern, as does question answering.

The influence of T5 reaches well beyond the model itself. Relative position biases, which T5 popularized for large-scale Transformers, became a standard component in many subsequent architectures. The span corruption pretraining objective demonstrated that masked language modeling could be made more effective by corrupting spans rather than individual tokens. And the text-to-text framing anticipated the instruction-following paradigm that would later dominate the field: if you can express any task as "here is your instruction, here is your input, now produce the output," then a single model can learn to follow arbitrary natural language directions.

Understanding T5 also gives you a clearer picture of the range of Transformer architectures. By the time T5 was released, the field had split into two main camps: encoder-only models like BERT, optimized for understanding tasks, and decoder-only models like GPT, optimized for generation. T5 offered a third path, one that maintained the encoder-decoder separation from the original Transformer and showed it could compete with specialized designs across a wide range of tasks. The story of why certain architectures suit certain tasks, and how training objectives shape the representations models learn, runs through every section of this chapter.