T5 Architecture › Limitations and Impact

T5's encoder-decoder architecture offers advantages for certain task types but introduces trade-offs compared to decoder-only alternatives. The bidirectional encoder excels when the full input must be processed before generating output. Summarization and translation benefit from understanding the complete context first; question answering does too. However, this architecture requires separate encoder and decoder computations, increasing memory requirements compared to decoder-only models of similar parameter counts. For the same total parameters, a decoder-only model dedicates all capacity to a single transformer stack, while an encoder-decoder splits parameters between two stacks. In practice, for a fixed parameter budget, decoder-only models often outperform encoder-decoder models on generation tasks because they can allocate all capacity to the generative direction.

The text-to-text framework, while elegant, has practical limitations. Classification tasks produce output tokens that must be mapped back to discrete labels, adding a parsing step that can fail if the model generates unexpected text. For high-throughput classification in production, task-specific heads on BERT-style models often prove more efficient. Additionally, regression tasks require outputting numbers as text strings, which is less numerically precise than dedicated regression heads. The model might output "3.7" when the correct answer is "3.71," a rounding error that would never occur with a regression head. Similarly, structured prediction tasks (like named entity recognition with overlapping spans) require serializing complex structures as text, which introduces format learning overhead and potential parsing errors.

T5 also shows the limitations of its training corpus. C4's English-centric, web-derived content means T5 performs best on the kinds of text that appear frequently on the web: news articles, encyclopedia-style text, and conversational English. Tasks requiring specialized domain knowledge (legal, medical, scientific) benefit from domain-adaptive pretraining or fine-tuning on domain-specific data. The filtering pipeline used to create C4, while effective at removing low-quality content, also removed content in non-English languages and may have inadvertently filtered out content from underrepresented communities whose writing style differs from mainstream web English.

Context length is another structural limitation. T5's relative position biases handle sequences of up to 512 tokens naturally, with graceful degradation beyond that. Many real-world tasks, summarizing a research paper, answering questions about a book chapter, processing long legal documents, exceed this limit. LongT5 addresses this directly, but the standard T5 architecture requires chunking or truncation strategies that can disrupt cross-chunk context.

T5 affected the field in several ways. The systematic ablation study in the original paper influenced many subsequent design decisions. Researchers could consult T5's experiments rather than re-running their own, with justified confidence that the findings generalized beyond the specific models tested. The text-to-text framework demonstrated that unified architectures could match or exceed task-specific approaches, paving the way for general-purpose instruction-following models. T5's pretraining recipe, combining span corruption with large-scale data, informed the development of models like PaLM and Flan-PaLM. Its relative position biases appeared in subsequent models from multiple research groups, often cited as superior to absolute positions for general-purpose language understanding.

T5 also established that encoder-decoder architectures remained competitive even as decoder-only models (the GPT family) gained prominence. This architectural diversity has proven useful. Encoder-decoder models continue to excel at translation and summarization, while decoder-only models dominate open-ended generation. Understanding both paradigms remains needed for practitioners choosing the right architecture for their specific application.