GLUE & SuperGLUE (2018) › Limitations

Despite their impact, GLUE and SuperGLUE faced limitations that became more apparent as the field progressed. The most fundamental concern was whether performance on these benchmarks truly reflected language understanding capabilities or merely sophisticated pattern matching. Models achieving high GLUE scores might recognize statistical patterns in the training data without grasping underlying linguistic structures or world knowledge. Success on the benchmark did not guarantee that systems could handle real-world language understanding tasks requiring commonsense reasoning, causal understanding, or context-sensitive interpretation.

The benchmarks' reliance on specific task formulations created potential for dataset artifacts and superficial patterns that did not generalize. Models might learn to exploit spurious correlations between surface-level features and labels rather than developing deep understanding. For example, a model might associate certain words with sentiment labels without understanding the semantic content of sentences. These dataset-specific patterns could produce high benchmark scores while failing to capture language understanding that would transfer to new domains or tasks.

Task selection in GLUE and SuperGLUE reflected specific choices about what constitutes language understanding, potentially missing important capabilities. The benchmarks emphasized classification and sentence-pair tasks but included limited evaluation of generation, dialogue, or multi-turn reasoning. Systems might excel at GLUE tasks while struggling with tasks requiring generating coherent text, maintaining conversation context, or reasoning over multiple documents. The narrow focus on specific task types meant that benchmark performance might not predict success in broader language understanding applications.

The aggregate scoring approach, while providing a convenient single metric, obscured important differences in per-task performance. A system might achieve a high aggregate score by excelling on easier tasks while performing poorly on more challenging ones. This aggregation could mask weaknesses in specific capabilities, making it difficult to identify which aspects of language understanding needed improvement. Researchers focused on aggregate scores might miss opportunities to address specific gaps in model capabilities.

Human baseline comparisons, while providing useful reference points, faced challenges in accurately estimating human performance. Crowdsourced annotation might not reflect the capabilities of expert linguists or domain specialists. Different annotators might disagree on challenging examples, and aggregation methods might not capture the full range of acceptable human judgments. The gap between machine and human performance might be smaller or larger than reported depending on how human baselines were established.

The benchmarks' focus on English-language tasks limited their applicability to multilingual systems or cross-lingual transfer scenarios. GLUE and SuperGLUE primarily evaluated English language understanding, potentially missing important capabilities in other languages or cross-lingual transfer. As the field moved toward multilingual models, the benchmarks provided limited guidance for evaluating these systems' capabilities across languages.

The rapid improvement in model performance created challenges for benchmark longevity. As models approached or exceeded reported human baselines, the benchmarks became less useful for distinguishing between systems or measuring progress. SuperGLUE addressed this by selecting more difficult tasks, but the fundamental issue remained: benchmarks that are too easy become saturated, while benchmarks that are too difficult might not reflect practical capabilities. Maintaining appropriate difficulty levels required ongoing updates and new task selection.

The evaluation protocols, while standardized, might not reflect real-world usage conditions. Training and evaluation data might have different distributions than actual application contexts. Models optimized for GLUE might not perform as well when deployed in production environments with different data characteristics, user populations, or task requirements. The benchmark's focus on clean, well-formatted text might not capture challenges in handling noisy, informal, or domain-specific language.