MMLU › Evaluation Protocol

The standard MMLU evaluation protocol specifies how models should be prompted and how their outputs should be scored. This protocol is important for comparable results across different models and research groups. Without strict standardization, differences in prompt formatting, example selection, or scoring logic could dominate actual capability differences, making leaderboard comparisons meaningless. The protocol has evolved through community consensus, with major evaluation frameworks like the EleutherAI Language Model Evaluation Harness and Hugging Face's evaluation libraries implementing these standards to ensure reproducibility.

MMLU is primarily evaluated in a few-shot setting, typically using 5 examples (5-shot) from the development set as context. As we explored in earlier chapters on in-context learning, few-shot prompting gives the model with examples of the task format without updating model parameters. This approach tests the model's ability to recognize patterns and adapt its behavior based on context, a capability that emerges strongly in large language models.

The choice of 5 examples is a practical compromise between giving sufficient context for the model to understand the task and fitting within context length constraints. With fewer examples, models might fail to recognize the multiple-choice format and generate free-form text rather than answer letters. With more examples, the context window fills, leaving less room for the actual question and potentially degrading performance due to attention dilution. Empirical testing across various model sizes has shown that 5 shots typically gives enough examples for models to infer the expected response format while leaving ample capacity for the target question.

For MMLU, the few-shot examples follow this template:

The prompt begins with an explicit statement of the task type, "The following are multiple choice questions (with answers)," which helps the model recognize the evaluation context. Each example follows a consistent structure: the question stem, the four labeled options, and the answer line. This consistency is important because it allows the model to learn the mapping between the task structure and the desired output format.

The model then generates a completion, and the first non-whitespace character is extracted and compared to the correct answer (A, B, C, or D). This extraction method is reliable to minor formatting variations, such as whether the model generates "B" or "B." or "B) ". However, it assumes that the model will generate the answer letter as the first token of its completion, which depends on the prompt ending with "Answer:" to trigger the expected continuation pattern.

An important subtlety arises here: some implementations extract the single-character answer by looking at the first token generated, while others score based on the log-probabilities that the model assigns to each of the four answer tokens (A, B, C, D). The log-probability approach is more reliable because it does not depend on the model generating a specific character; instead, it directly measures how much probability mass the model assigns to each answer option. However, it requires access to the model's internal probability distributions, which is not always available through API-based evaluation. The two approaches can yield substantially different scores on the same model, adding another source of incomparability between published results.

MMLU accuracy is computed as the simple percentage of questions answered correctly across all subjects. However, there are important nuances in how this aggregation occurs that materially affect interpretation of results.

Subject-Level Accuracy: Each subject's accuracy is computed independently. This allows analysis of performance across domain categories, such as comparing STEM versus humanities performance. Subject-level analysis is needed for diagnosing specific weaknesses. For example, a model might reach high overall accuracy while failing on Professional Medicine or Abstract Algebra, showing safety concerns for medical applications or limitations in mathematical reasoning.

Macro-Averaging: The overall MMLU score is typically computed as the average of per-subject accuracies, giving equal weight to each subject regardless of the number of questions it contains. This prevents large subjects from dominating the aggregate score. For instance, High School Mathematics might contain more questions than World Religions, but macro-averaging ensures both contribute equally to the final metric.

More formally, if we denote the accuracy on subject $i$ as $a_i$ and there are $S = 57$ total subjects, then the macro-averaged MMLU score is:

$$\text{MMLU}_{\text{macro}} = \frac{1}{S} \sum_{i=1}^{S} a_i$$

where each $a_i$ is itself computed as:

$$a_i = \frac{\text{correct}_i}{\text{total}_i}$$

Here, $\text{correct}_i$ denotes the number of correctly answered questions in subject $i$ and $\text{total}_i$ denotes the total questions in subject $i$.

Micro-Averaging: Alternatively, some implementations compute accuracy across all questions globally. If the total number of questions across all subjects is $N = \sum_{i=1}^{S} \text{total}_i$, then the micro-averaged score is:

$$\text{MMLU}_{\text{micro}} = \frac{\sum_{i=1}^{S} \text{correct}_i}{N}$$

This approach weights subjects by their question count, meaning that subjects with more test questions have greater influence on the final score. While micro-averaging shows the raw probability of answering a randomly selected question correctly, it can obscure performance on rare but important domains.

The choice between macro and micro averaging is not merely technical. Macro-averaging shows the philosophical stance that broad competence requires mastery across all domains, treating a failure on World Religions the same as a failure on High School Chemistry. Micro-averaging shows a statistical stance: it estimates the probability that a randomly drawn question from the benchmark will be answered correctly. In practice, most published MMLU scores use macro-averaging across subjects, which is why you should verify the aggregation method when comparing results across papers.

While 5-shot evaluation is standard, researchers also report zero-shot performance to assess a model's inherent knowledge without task-specific formatting. The gap between zero-shot and few-shot performance indicates how effectively a model can adapt to task structure from examples.

Zero-shot: The model sees only the target question without examples. This tests raw knowledge and the ability to infer the task from instructions. Zero-shot performance reveals what the model has learned during pre-training, independent of its ability to recognize evaluation formats. However, zero-shot results can be artificially depressed if the model generates well-reasoned answers in a format that does not match the expected single-letter extraction pattern. A model might write "The answer is B because..." in zero-shot mode, which would be scored as incorrect by a naive letter-extractor.

Few-shot (5-shot): The model sees 5 examples before the target question. This tests in-context learning capability and robustness to formatting. Few-shot performance typically exceeds zero-shot performance because the examples clarify the expected output format and may trigger relevant knowledge associations through the exemplars provided.

Large models often show large improvements from zero-shot to few-shot settings, sometimes gaining 10-20 percentage points. This gap shows their ability to recognize and adapt to examination formats. The magnitude of this improvement varies by domain. Technical subjects with clear right answers often show smaller gaps, as the model either knows the fact or does not, regardless of format. Complex reasoning tasks may show larger gaps, as the examples help the model recognize the depth of analysis required. Analyzing the zero-shot to few-shot gap across subjects gives insight into which domains benefit most from in-context learning versus which require explicit knowledge acquisition during training.

There is a deeper interpretive question lurking here. If a model improves substantially from zero-shot to few-shot, does that improvement stand for real learning from the examples, or does it stand for the model recalibrating its output format while accessing the same underlying knowledge? The distinction matters for interpreting what MMLU measures. If format recalibration accounts for most of the few-shot gain, then zero-shot performance may be the more honest measure of actual knowledge, while few-shot performance may better reflect deployment performance where users give context.

Beyond the simple letter-extraction approach, a more principled scoring method relies on the conditional probabilities that the model assigns to each answer option. Rather than asking the model to generate a completion and then parsing the first character, this approach queries the model for the log-probability it assigns to each of the four answer tokens given the prompt.

For a given question $q$ with answer choices $\{A, B, C, D\}$, the probability-based approach selects the answer $\hat{y}$ that maximizes the conditional probability:

$$\hat{y} = \arg\max_{y \in \{A, B, C, D\}} \log P(y \mid \text{prompt}_q)$$

where $\text{prompt}_q$ is the full few-shot context plus the question with "Answer:" appended.

This method has several advantages over generation-based scoring. It is deterministic, regardless of temperature or sampling settings. It does not depend on the model generating a specific character, eliminating errors where the model generates a longer response like "The answer is B" and the parser fails to extract the intended letter. It also allows computation of confidence scores: the difference between the highest and second-highest log-probabilities indicates how decisively the model favors one answer over the others.

However, probability-based scoring is not without complications. The log-probability of a single token depends on how the model's tokenizer is that token. For most models, "A", "B", "C", and "D" are single tokens and their probabilities are directly comparable. But some models might tokenize these differently, or the probability of "A" might be systematically higher or lower than "B" due to training data distributions where certain letters are more common in certain contexts. Researchers sometimes apply a normalization step to account for these systematic biases before comparing probabilities across the four options.