GLUE & SuperGLUE (2018) › The Problem

Before GLUE, evaluating language understanding systems suffered from fundamental fragmentation that prevented meaningful comparisons and slowed progress. Each research group would select different tasks to evaluate their models, use different datasets even for the same task, apply different preprocessing and evaluation protocols, and report results in different formats. A paper might claim state-of-the-art performance on sentiment analysis, but readers had no way to know how that system would perform on question answering, textual entailment, or other language understanding tasks. This fragmentation made it difficult to assess whether an approach improved language understanding across tasks or merely optimized one benchmark.

The lack of standardized evaluation created several specific problems. First, it was nearly impossible to compare results across papers. One group might report accuracy on a sentiment classification task, while another might report F1 scores on a different sentiment dataset. Without shared datasets and protocols, readers could not determine which system was truly better. Second, researchers tended to optimize their models for individual tasks, creating highly specialized systems that performed well on one problem but failed to generalize. A model excelling at sentiment analysis might perform poorly on question answering, suggesting it had learned task-specific patterns rather than general language understanding.

The evaluation field also lacked diversity in task types. Many evaluation efforts focused on a narrow set of tasks like classification or sequence labeling, missing important capabilities like natural language inference, semantic similarity, or commonsense reasoning. A system might appear impressive when tested only on sentiment classification, but fail when confronted with tasks requiring understanding of logical relationships, coreference resolution, or multi-sentence reasoning. Without evaluation across diverse task types, researchers could not assess whether their systems possessed reliable language understanding or merely superficial pattern matching capabilities.

The reporting of results created additional confusion. Papers might use different metrics for the same task, making comparisons difficult. Some papers reported performance on development sets, others on test sets, and still others on custom splits. Training data varied widely, with some systems using task-specific training data while others used additional unlabeled data or transfer learning. These inconsistencies meant that apparent improvements might reflect better training procedures or data rather than architectural or algorithmic advances. Readers struggled to separate new methods from methodological variations.

For researchers developing pre-trained language models, the lack of evaluation across tasks was particularly problematic. These models were explicitly designed to learn general language representations that could transfer across tasks. However. This showed this transfer required evaluating on multiple diverse tasks using consistent protocols. Without a benchmark like GLUE, researchers had to manually select tasks, prepare datasets, implement evaluation protocols, and report results, consuming significant time and introducing opportunities for inconsistency. This overhead discouraged evaluation across a broad task set and limited the field's ability to assess whether pre-training approaches were delivering on their promise of general language understanding.

The field also lacked clear criteria for what constituted human-level performance or what represented meaningful progress. Without a standardized benchmark with established human baselines, it was difficult to determine how far systems were from human capabilities or which improvements transferred beyond a benchmark rather than reflecting incremental optimization. The absence of a unified evaluation framework meant that the field lacked clear milestones and progress indicators that could guide research directions and resource allocation.