Evaluation Challenges › Evaluation Brittleness

Even when contamination is not present, evaluation results can be surprisingly fragile. A model may score 90% on one version of a task and 65% on a semantically equivalent reformulation. This brittleness is not a quirk; it is a systematic property of how neural networks represent and process language.

The reason for brittleness lies in how language models process input. A transformer model does not extract abstract meaning from a prompt and then reason about that meaning in isolation. It processes a specific sequence of tokens, and the statistical patterns associated with those tokens influence every downstream computation. When you change the surface form of a prompt, even while preserving its logical content, you change the token sequence and thereby shift the internal activations. For a robust reasoner, these shifts would be small and well within the range where the correct answer can still be reached. For a system whose performance depends on pattern matching, even small surface changes can push the model into a different region of its parameter space where different outputs are favored.

Language model performance depends heavily on how tasks are phrased. The same question, posed in slightly different ways, can produce dramatically different accuracy. This sensitivity is well-documented and has several causes.

Format effects: Models trained on instruction-following data learn associations between formatting conventions and task types. A model may perform better on multiple-choice questions when the choices are labeled A, B, C, D than when they are labeled 1, 2, 3, 4, or Option A, Option B, not because the underlying knowledge differs, but because the familiar format activates the relevant processing patterns. The model has seen many multiple-choice questions in its training data formatted in certain ways, and those formats serve as retrieval cues.

Instruction framing: "Answer in one word" versus "Provide a brief answer" versus "Reply concisely" can all produce different token distributions, which in turn affect which answer the model selects. Even apparently minor phrasing differences, like whether instructions appear before or after the question, affect performance. This sensitivity to instruction framing is a reflection of how instruction-tuned models were trained: they learned to follow certain linguistic patterns, and small deviations from those patterns may push the model toward less reliable outputs.

Few-shot example selection: When evaluations use few-shot prompting (providing example question-answer pairs before the actual question), the choice of examples matters enormously. Using easier or harder examples, or examples from a different distribution, can shift performance by 10 to 20 percentage points on some benchmarks. The model uses the few-shot examples to infer the task format and the expected response style, so examples that suggest the wrong format or calibration will lead it astray.

Chain-of-thought sensitivity: Models often perform better when prompted to reason step by step before producing an answer. But the specific incantation matters. "Let's think step by step" works differently from "First, let me reason through this" on the same model, even though the semantic intent is identical. This is partly an artifact of training data: specific phrases that appear frequently in reasoning demonstrations are more likely to trigger the reasoning behavior the model learned to associate with them.

Positional and ordering effects: The position of the correct answer among multiple-choice options affects accuracy. If a model has a bias toward selecting certain answer positions (say, "B" or "C" rather than "A" or "D"), shuffling answer choices will change its measured performance even though the underlying question and correct answer remain the same. Several studies have documented that this positional bias is substantial for some models, and that standard benchmark evaluations that fix answer positions are systematically affected.

This means that a benchmark result is more than a measure of a model's capability; it is a measure of a model's capability combined with the specific prompt format used in the evaluation. Two research papers using slightly different prompt formats may measure different things and reach different conclusions about the same model.

The items included in a benchmark shape what gets measured in subtle ways. Even well-designed benchmarks have compositional properties that can be exploited or that may not generalize.

Topic and difficulty distribution: A reading comprehension benchmark might oversample texts from one domain (news articles, for example) and undersample others (scientific papers, legal documents). A model that has heavily trained on news text may outperform on such a benchmark not because it is a better reader in general, but because it is better in the evaluated domain. When aggregate scores hide domain-specific variation, they can create misleading impressions of general capability.

Label balance: In classification tasks, the distribution of correct answers matters. If a multiple-choice benchmark happens to have "B" as the correct answer 40% of the time, a model with a bias toward selecting "B" will appear more capable than it is. This has been observed in practice: some models develop positional biases in multiple-choice settings. The MMLU benchmark, for example, has unequal answer distributions across subjects, which means models with different positional biases will perform differently across subjects for reasons unrelated to knowledge.

Spurious correlations: Benchmark items sometimes contain unintended cues that allow high performance without the target skill. Natural language inference datasets have been found to contain lexical patterns (certain words appearing more often in entailment examples, for instance) that allow models to classify without performing the intended reasoning. A model can achieve high accuracy on such a benchmark by exploiting these surface cues while failing entirely on items that require the reasoning the benchmark claims to measure. The celebrated fact that BERT-style models achieved near-human performance on NLI benchmarks was followed by discovery that much of this performance came from exploiting dataset artifacts rather than reasoning about entailment.

Length and complexity biases: Items that are shorter, simpler, or more stylistically uniform may be easier to answer through pattern matching than items that require sustained attention and complex reasoning. A benchmark that inadvertently over-represents simple items will overestimate a model's capability on difficult ones. Similarly, if benchmark items share a characteristic style (because they were written by the same authors or drawn from the same source), models that encountered similar text in training will benefit from style familiarity.

Beyond prompt format, the expected output format shapes results. Models evaluated with free-form generation often perform differently from the same models evaluated with constrained generation (selecting from a fixed list of choices). This is not surprising; these are different tasks. But it complicates comparison across evaluations that use different output formats.

Open-ended generation evaluation introduces additional brittleness through the evaluation process itself. If a model says "Paris" to a geography question and the expected answer is "Paris, France," does the model get credit? Exact match metrics will penalize this, while ROUGE or BERTScore (covered in earlier chapters of this handbook) will give partial credit. The choice of output scoring methodology becomes part of the evaluation result. Two evaluators using the same model and the same benchmark may reach different conclusions about its accuracy if they use different answer extraction and scoring heuristics.

This problem is particularly acute for open-ended generation tasks like summarization, translation, and question answering with long answers. For these tasks, no single automated metric captures all dimensions of quality, and different metrics often disagree on which outputs are best. Fluency and factual accuracy are different properties that need separate measurement; a model can produce fluent, confident summaries that are factually wrong, and no single score captures this distinction.