T5 & Text-to-Text Framework (2019) › Limitations

Despite its significant achievements, T5 had important limitations that would shape subsequent research directions. One fundamental limitation was computational cost. The encoder-decoder architecture required running both an encoder and a decoder for every task, even for tasks like classification that might have been handled more efficiently with encoder-only models. The encoder processed the input, and the decoder generated the output, requiring roughly twice the computational resources of a decoder-only model for generation tasks or an encoder-only model for understanding tasks.

The span corruption pre-training objective, while effective, also had limitations. The objective required generating entire spans during pre-training, which could be slower than masked language modeling that only needed to predict individual tokens. This made pre-training more computationally expensive. Additionally, the span corruption objective might not be optimal for all downstream tasks. Tasks that required understanding individual tokens or fine-grained linguistic analysis might have benefited more from token-level objectives like masked language modeling.

The text-to-text framework, while elegant, also had drawbacks. Reframing classification tasks as text generation could be less efficient than direct classification. A task that naturally had a small number of classes would require the model to generate text and then match it against possible classes, rather than directly outputting a class probability distribution. This could make inference slower and less accurate for pure classification tasks compared to specialized classification models.

The reliance on task prefixes also introduced challenges. The model needed to learn the meaning of each prefix during training, which required exposure to examples with that prefix. For new tasks or tasks with limited training data, the model might not understand novel prefixes well, limiting zero-shot or few-shot capabilities. While fine-tuning could address this, it required task-specific training data, reducing some of the benefits of the unified framework.

The model size and training data requirements were also limitations. Achieving state-of-the-art performance required training very large models on massive datasets. The T5-11B model, which achieved the best results, required substantial computational resources for both training and inference. This limited access to the best-performing models and made it difficult for smaller organizations or researchers to replicate results or build on the work.

The C4 dataset, while large and diverse, also had limitations. The dataset was created from web crawl data, which meant it reflected biases and limitations present in web content. The dataset might have underrepresented certain languages, dialects, or domains, limiting the model's capabilities in those areas. Additionally, web data could contain misinformation, biased content, or problematic material that would be learned by the model, creating ethical concerns.

Another limitation was that the text-to-text framework might not be optimal for all tasks. Some tasks had inherent structure that specialized architectures could model more effectively. For example, tasks involving structured output like parsing or knowledge graph construction might benefit from architectures that could explicitly model that structure. The text-to-text framework's flexibility came at the cost of not being able to incorporate task-specific inductive biases.

The evaluation challenges also persisted. While the unified framework simplified some aspects of evaluation, many tasks still required task-specific metrics that couldn't be easily unified. Translation used BLEU scores, summarization used ROUGE, and question answering used F1 or exact match. These metrics measured different aspects of performance and weren't directly comparable, making it difficult to assess overall progress or understand trade-offs between tasks.