MMLU › Summary

MMLU (Massive Multitask Language Understanding) is a measurable shift in language model evaluation from narrow single-task metrics to complete knowledge assessment across 57 subjects spanning STEM, humanities, social sciences, and professional domains. Introduced by Hendrycks and colleagues in 2020, the benchmark tests models using multiple-choice questions ranging from elementary to professional difficulty levels, with standard evaluation protocols specifying 5-shot prompting and macro-averaged accuracy across subjects.

The technical structure of MMLU lets clean, reproducible evaluation through its four-option multiple-choice format, though this format tests recognition rather than generation capabilities. The strict formatting allows for automated evaluation without human judges. This makes possible rapid iteration and comparison across dozens of models. Evaluation involves formatting few-shot examples from development sets, prompting models to generate answer continuations or computing log-probabilities over answer tokens, and aggregating per-subject and overall accuracies. The macro-averaging formula $\text{MMLU}_{\text{macro}} = \frac{1}{S}\sum_{i=1}^{S} a_i$ treats all 57 subjects equally. This keeps rare domains contribute as much as common ones.

The evaluation protocol matters as much as the questions themselves. The choice between generation-based and probability-based scoring, the number of few-shot examples, the random seed for development set selection, and even the formatting of answer option labels can all shift reported scores by several percentage points. These protocol sensitivities mean that comparing MMLU scores across papers requires careful attention to implementation details.

Critical limitations include potential training contamination from publicly available test sources, cultural and linguistic biases toward Western educational contexts, format constraints that favor recognition over reasoning, and decreasing discriminative power as models approach human-level performance. Prompt sensitivity and few-shot example variance add measurement uncertainty that complicates direct model comparisons. Perhaps most importantly, MMLU does not measure calibration, open-ended generation, multi-step reasoning transparency, or factual consistency, all of which matter greatly for real-world deployment.

When interpreted within these constraints, MMLU is a useful diagnostic tool for identifying knowledge gaps across domains and tracking broad progress in language model development. As we will explore in upcoming chapters on commonsense reasoning and mathematical benchmarks, modern evaluation requires a portfolio of benchmarks targeting distinct aspects of intelligence beyond factual knowledge retrieval.