BIG-bench & MMLU (2023) › 2023: BIG-bench and MMLU

The year 2023 marked a watershed moment in the evaluation of large language models when researchers recognized that existing benchmarks had become insufficient for assessing the rapidly expanding capabilities of modern AI systems. While benchmarks like GLUE, SuperGLUE, and SQuAD had driven significant progress in the late 2010s and early 2020s, the emergence of models with hundreds of billions of parameters revealed fundamental limitations in how the field measured language understanding. Two landmark benchmarks, BIG-bench (Beyond the Imitation Game Benchmark) and MMLU (Massive Multitask Language Understanding), emerged to address these limitations by expanding evaluation across broader reasoning capabilities, diverse knowledge domains, and specialized tasks that better reflected the full scope of language model capabilities.

BIG-bench emerged from a collaborative effort involving hundreds of researchers across dozens of institutions, making it one of the largest evaluation efforts in AI history. Released in 2022 and gaining widespread adoption in 2023, BIG-bench assembled over 200 diverse tasks covering topics from mathematics and physics to literature analysis and social reasoning. Unlike previous benchmarks that focused on narrow NLP tasks, BIG-bench explicitly sought to test capabilities that went beyond traditional language understanding, including reasoning, knowledge application, and creative problem-solving.

MMLU, developed by researchers at UC Berkeley and introduced in 2020 but reaching prominence in 2023, took a different but complementary approach. Instead of creating new tasks, MMLU assembled standardized academic and professional examinations covering subjects from high school mathematics to professional law and medicine. The benchmark included 57 tasks across four major domains: humanities, social sciences, STEM, and other professional subjects. By using real academic examinations, MMLU tested whether language models could apply their training knowledge across the breadth of human knowledge domains. This provided a more realistic assessment of how well these systems understood and could apply information across specialized fields.

the effect of these benchmarks extended far beyond providing additional evaluation tasks. They changed how the field thought about language model capabilities. Previous benchmarks had treated language understanding as a relatively narrow domain of NLP tasks. BIG-bench and MMLU reframed the question: could language models demonstrate broad reasoning, specialized knowledge, and problem-solving capabilities that approached human-level performance across diverse domains? This perspective aligned with the growing realization that large language models had become general-purpose reasoning engines, not merely text processing systems, and needed evaluation accordingly.

These benchmarks arrived when the field needed harder tests. By 2023, models like GPT-4, Claude, and their successors were achieving human-level or near-human performance on many traditional NLP benchmarks. GLUE and SuperGLUE scores had reached or exceeded reported human baselines on multiple tasks. This success raised an important question: had language models truly achieved human-level language understanding, or had the benchmarks simply been optimized past their usefulness? BIG-bench and MMLU provided fresh challenges that better reflected the complexity and breadth of capabilities that language models were demonstrating.