MMLU › The Structure and Scope of MMLU

MMLU is organized as a collection of 15,908 questions drawn from a variety of standardized tests, professional exams, and academic subjects. The questions are divided into 57 distinct tasks covering elementary mathematics, US history, computer science, law, and more. This diversity ensures that high performance requires broad knowledge and reasoning capabilities rather than expertise in a single domain. The breadth of the benchmark serves an important methodological purpose: it prevents models from overfitting to specific question styles or narrow knowledge bases. Just as a student who memorizes answers to one specific test has not truly mastered a subject, a model that excels only on chemistry or only on history has not demonstrated general language understanding. The multitask structure forces models to maintain distinct reasoning frameworks for different types of knowledge, switching between mathematical proof, historical analysis, and legal interpretation as the context demands.

The dataset is divided into training, validation (development), and test splits. The training split is relatively small and primarily useful for supervised fine-tuning experiments. The validation split contains five examples per subject, which is precisely the number used for 5-shot evaluation, making the split design deliberate rather than arbitrary. The test split forms the primary evaluation target and contains between 100 and several hundred questions per subject, depending on how many practice questions were available for that domain. The total of 15,908 questions is only the test and validation examples, not the much smaller training set.

This split design shows an important methodological choice. By giving exactly five development examples, the benchmark authors anticipated that researchers would use those examples as few-shot demonstrations. The alignment between the development set size and the standard few-shot count (5) means that a 5-shot evaluation always uses the entire development set. This keeps consistency across implementations. If the development set were larger, researchers might make different choices about which examples to select, introducing another source of variance.

The 57 subjects are organized into four high-level categories, each testing different aspects of knowledge and reasoning. This taxonomy shows the traditional organization of academic and professional knowledge, letting researchers to analyze aggregate performance and the distribution of capabilities across different cognitive domains. The categorization also helps identify systematic weaknesses in models, such as whether they struggle more with formal reasoning than with factual recall, or whether cultural biases affect performance on humanities versus STEM subjects.

STEM (Science, Technology, Engineering, Mathematics)

These subjects test formal reasoning and factual knowledge in technical domains. Success in these areas requires the ability to manipulate abstract symbols, apply mathematical theorems, understand physical laws, and follow logical derivations. The STEM category includes subjects such as:

• Abstract Algebra, Astronomy, Biology, Chemistry, College Mathematics, Computer Science, Conceptual Physics, Electrical Engineering, Elementary Mathematics, Formal Logic, High School Biology, High School Chemistry, High School Computer Science, High School Mathematics, High School Physics, High School Statistics, Machine Learning, Physics

These subjects vary considerably in their reasoning demands. While Elementary Mathematics tests arithmetic and basic algebraic manipulation, Abstract Algebra requires understanding group theory, rings, and fields. Similarly, Conceptual Physics tests qualitative understanding of physical principles, while College Mathematics demands rigorous proof-based reasoning. This gradient of difficulty within STEM allows MMLU to discriminate between models at different capability levels.

The distinction between "High School" and "College" variants of the same subject matters. High School Chemistry might ask about atomic orbitals and basic stoichiometry, while a college-level chemistry question might probe thermodynamic equilibria or reaction mechanisms. The existence of multiple difficulty levels within a subject domain gives fine-grained evidence about where models fall on the competency spectrum.

Humanities

These subjects evaluate cultural knowledge, interpretive skills, and historical understanding. Unlike STEM subjects, where answers are often deterministically correct based on logical derivation, humanities questions frequently require fine-grained interpretation of texts, understanding of historical context, and appreciation of cultural frameworks. The humanities category includes:

• High School European History, High School US History, High School World History, Philosophy, Prehistory, Professional Law, World Religions, Formal Logic (overlaps with STEM)

The inclusion of philosophy and world religions tests whether models can engage with abstract ethical reasoning and comparative cultural analysis, capabilities that are needed for safe deployment in diverse social contexts. Professional Law, while practical in application, requires interpreting legal texts within historical and precedent-based frameworks, blending factual recall with hermeneutic skills.

Historical subjects deserve particular attention because they test a form of contextual knowledge that differs from scientific facts. Knowing that the Treaty of Versailles was signed in 1919 is a date-retrieval task, but understanding why its terms contributed to the conditions that enabled World War II requires synthesizing economic, political, psychological, and social knowledge into a coherent causal narrative. MMLU questions in history often operate at this higher level of analysis.

Social Sciences

These subjects assess understanding of human behavior, institutions, and economic systems. They occupy a middle ground between the formal precision of STEM and the interpretive flexibility of humanities, often requiring both factual knowledge and theory application. The social sciences category includes:

• Anatomy, Business Ethics, Clinical Knowledge, College Medicine, Economics, Global Facts, High School Geography, High School Government and Politics, High School Macroeconomics, High School Microeconomics, High School Psychology, Human Aging, Human Sexuality, International Law, Judeo-Christian Ethics, Logical Fallacies, Marketing, Medical Genetics, Miscellaneous, Nutrition, Professional Accounting, Professional Law (overlaps with Humanities), Professional Medicine, Public Relations, Security Studies, Sociology, US Foreign Policy, Virology

The clinical and medical subjects within this category deserve special attention. Unlike textbook biology questions, Clinical Knowledge and Professional Medicine present scenarios that require differential diagnosis, understanding of disease progression, and awareness of treatment protocols. These questions test whether models can apply biological facts to specific patient presentations, a form of reasoning that closely parallels medical decision-making. A question might present vital signs, laboratory values, and a brief history, then ask for the most likely diagnosis or the appropriate next step in management. Success requires factual recall of disease definitions and an understanding of how diseases present, how they progress, and how clinicians discriminate between similar presentations.

Economics questions similarly require applying theory to scenarios. Knowing the definition of price elasticity is not enough; MMLU economics questions might ask what happens to consumer surplus when a price floor is introduced above the equilibrium price, requiring the student to trace through supply and demand dynamics to a specific quantitative or qualitative conclusion.

Other

This category captures practical and professional knowledge domains that do not fit neatly into the previous classifications but stand for important real-world competencies. It includes subjects like Global Facts, Miscellaneous, and security-related topics that draw from practical knowledge rather than formal academic disciplines. The "Miscellaneous" subject is particularly interesting because it draws from diverse general knowledge sources, testing breadth of awareness rather than depth in any single domain.

Every question in MMLU follows a strict multiple-choice format with exactly four options labeled A, B, C, and D. This structure serves several important purposes beyond simple scoring convenience. First, the format mirrors standardized testing practices used in human education systems worldwide. This gives an intuitive baseline for comparing model performance to human expertise. Second, the constraint of exactly four options creates a known random baseline of 25% accuracy. This makes it easy to assess whether models are performing clearly above chance. Third, the format allows for the construction of advanced distractors, incorrect options that are plausible but subtly wrong.

The use of four options is a careful balance. Too few options would make random guessing too successful and reduce the discriminative power of the test. Too many options might introduce parsing errors where models confuse option letters or struggle to maintain all possibilities in working memory, confounding reasoning ability with format handling. Four options gives enough complexity to test real discrimination while remaining cognitively manageable.

The quality of distractors is what separates a good multiple-choice question from a trivial one. In well-designed MMLU questions, the three incorrect options each stand for a coherent but wrong answer that a student with partial knowledge might select. In a medical question, distractors might be real diseases with overlapping symptom profiles. In a history question, distractors might be real events from the same period that are plausibly connected to the question's context. In a mathematics question, distractors might correspond to common algebraic errors or sign mistakes. The careful construction of distractors means that selecting the correct answer requires real mastery rather than merely identifying absurd options.

The format lets several types of evaluation:

• Unambiguous evaluation: Unlike open-ended generation tasks where correctness might be debated, multiple-choice questions have objectively correct answers. This eliminates the need for human judges or complex automatic evaluation metrics that might introduce their own biases.

• Calibration testing: The format allows researchers to measure whether model confidence aligns with accuracy, though MMLU itself does not require confidence scores. Researchers can extend the basic protocol to ask models to report their certainty, letting analysis of whether models know when they are guessing versus when they are certain.

• Distractor analysis: The three incorrect options are designed to be plausible, testing whether the model can distinguish subtle distinctions in knowledge. Good distractors target common misconceptions or similar concepts. Analyzing which distractors models choose reveals the nature of their confusions and knowledge gaps.

Questions range in difficulty from elementary school level, such as basic arithmetic or historical facts, to graduate-level professional examinations involving complex legal reasoning or advanced medical diagnostics. This range lets MMLU to discriminate between models of varying capabilities, from small models that might struggle with high school material to large models that approach professional expertise. The difficulty gradient also allows researchers to identify "emergence points," specific capability thresholds where larger models suddenly master domains that smaller models cannot handle. As we discussed in the chapter on emergent capabilities, this kind of non-linear improvement is a hallmark of scale in language model development.

Understanding how the MMLU questions were collected illuminates both the benchmark's strengths and its limitations. The authors gathered questions from publicly available sources: practice exams used in university courses, free online study materials for professional licensing exams (such as the USMLE for medicine and the bar exam for law), and open educational repositories. They deliberately avoided proprietary question banks to ensure the benchmark could be used freely by researchers worldwide.

This sourcing strategy has an important consequence: the questions were not written specifically for the benchmark but rather adapted from existing educational materials designed for human students. This means the questions reflect the actual cognitive challenges that educators and examination boards have deemed appropriate for assessing competency at each educational level. A question from a professional medicine study guide was crafted by medical educators who understand what knowledge distinguishes a competent clinician from an incompetent one. This authenticity makes MMLU questions more real as tests of actual expertise than synthetically generated questions would be.

The authors made deliberate quality control choices during collection. They excluded questions that required images or tables to answer, as these would complicate text-only model evaluation. They also filtered questions with ambiguous or disputed answers, though some ambiguity inevitably remained. Questions requiring calculation were retained, as arithmetic is an important cognitive capability. The filtering process aimed to produce a benchmark where a knowledgeable human could reliably identify the correct answer with high confidence, establishing a real human baseline.

The resulting questions span a range from direct recall (what year did X happen?) to applied reasoning (given these findings, what is the diagnosis?) to analytical comparison (which of the following arguments contains a logical fallacy?). This spectrum of cognitive demands, drawn from Bloom's taxonomy of educational objectives, makes MMLU a more complete test than a pure fact-retrieval benchmark would be.