LLM Evaluation Fundamentals: Goals, Design, and Metrics

You have spent months training a large language model. The loss curves looked good. The model generates fluent text. But is it useful? Does it answer questions accurately, follow instructions reliably, or summarize documents faithfully? You need a way to measure this, and that is what evaluation gives you.

Evaluation is the systematic process of measuring how well a language model performs on tasks that matter. It is not a single activity but a collection of methodologies, each suited to different questions and different stages of development. A model that achieves low perplexity on held-out text might still generate factual errors. A model that scores highly on a multiple-choice benchmark might still struggle with open-ended generation. A model that impresses human annotators in lab settings might behave differently in deployment. Understanding which evaluation approach to apply, when, and how to interpret the results is one of the most practically important skills in working with language models today.

This chapter establishes the conceptual foundations for evaluation: why we evaluate, what we are trying to measure, how evaluation studies are designed, and what the main classes of metrics offer. We will also examine common failure modes in evaluation that can mislead practitioners who rely on them uncritically. The subsequent chapters in this part build on these foundations by examining specific metric families: automatic metrics, human evaluation, LLM-as-judge, and benchmark design.

The intuitive case for evaluation is obvious: you need to know if your model works. But there are more precise reasons why rigorous evaluation matters in practice, and understanding them shapes how you design your evaluation pipeline.

Training a language model optimizes a specific objective, typically next-token prediction loss or a reward model score during RLHF fine-tuning. These training objectives are proxies for what we care about: real-world usefulness. As we discussed in the Alignment and RLHF section, the gap between what we can measure computationally and what we want from a model is a central challenge. Evaluation is how we measure that gap.

When a model's training loss decreases, it means the model is getting better at predicting its training data. This often correlates with better task performance, but not always. A model trained on internet text will learn to produce fluent prose, but fluency does not imply accuracy. A reward-optimized model might learn to produce outputs that score highly without satisfying the underlying human intent, a phenomenon called reward hacking. Evaluation on diverse, independent test sets reveals these failures that training metrics cannot.

The proxy problem is not a temporary limitation that will be resolved by better metrics. It is structural. Any computable metric is, by definition, a simplification of the full complexity of real-world usefulness. Perplexity measures prediction accuracy on held-out text, not whether the model tells the truth. ROUGE measures token overlap, not whether a summary captures the most important information. Human preference ratings measure whether annotators liked a response, not whether the response was safe or accurate. Each metric is informative within its domain and uninformative outside it. The practical consequence is that no single metric can serve as the final arbiter of model quality. Evaluation is always a portfolio of signals, and part of the skill in evaluation design is knowing which signals matter for which questions.

Large model development involves many engineering decisions: architecture changes, data curation choices, hyperparameter sweeps, post-training steps like instruction tuning and RLHF. Each change can improve some behaviors while degrading others. A systematic evaluation suite tracks these changes over time, so you can detect regressions early rather than discovering them after deployment.

This is more subtle than it sounds. Evaluation suites must be broad enough to catch unexpected regressions in unrelated capabilities. A change to the instruction-tuning data that improves coding performance might inadvertently hurt math reasoning. Only broad evaluation catches this.

Regression detection is particularly important in post-training. Full fine-tuning or RLHF on a narrow instruction set can cause catastrophic forgetting of capabilities the base model had. A model fine-tuned to be a helpful coding assistant may degrade on language understanding tasks it handled well before fine-tuning. Without an evaluation suite that covers both the fine-tuned capabilities and the broader set of preserved capabilities, this regression will go undetected until users encounter it.

Continuous evaluation also enables longitudinal tracking of capability improvements, which is valuable for planning and for communicating progress to stakeholders. A team that logs evaluation results systematically across model versions can show exactly which capabilities improved as the training pipeline matured, which helps justify engineering investments and informs priorities for future work.

When multiple models or multiple training runs are available, evaluation provides the basis for selection. This requires careful thinking about what dimensions to compare across and whether your evaluation suite represents the actual deployment use case. Benchmark performance and deployment performance often diverge significantly, for reasons we will examine in detail.

The selection problem is harder than it appears. Consider a development team choosing between model A, which scores 2 points higher on a reasoning benchmark, and model B, which scores 3 points higher on a language understanding benchmark. Which model should go to production? The answer depends entirely on the relative importance of those capabilities for the actual use case. Without clarity on the deployment goals and without evaluation data from that distribution, the comparison is essentially arbitrary.

This is why evaluation-driven development requires defining success criteria before building evaluation pipelines. Teams that define what they are optimizing for before running evaluations make better decisions than teams that collect many metric scores and interpret them post-hoc. The evaluation suite should reflect the product requirements, not the other way around. Choosing a model solely based on the highest aggregate benchmark score is equivalent to selecting a car based on its horsepower alone, ignoring safety ratings, fuel efficiency, and cargo space.

Model selection also requires attention to evaluation variance. Stochastic decoding, random seeds in evaluation code, and sampling variability in test sets all introduce noise into metric values. A model that appears 1 point better than a competitor might be 1 point worse under different random seeds. Running multiple evaluation passes and reporting confidence intervals, rather than single-run point estimates, is the only way to make reliable selection decisions.

Evaluation results communicate model capabilities to stakeholders who cannot interact with the model directly. Published benchmark scores allow researchers to situate their work within the broader literature. They allow deployment teams to make informed decisions. They allow regulators and policymakers to assess safety and capability claims. This communication function creates strong pressure toward evaluation designs that are transparent, reproducible, and comparable across systems.

The accountability dimension of evaluation is particularly important as language models are deployed in high-stakes contexts. A medical question-answering system, a legal document analysis tool, or a content moderation pipeline all require evidence that the system performs reliably before deployment. Rigorous evaluation is not just good engineering practice in these settings; it is an ethical obligation. The people affected by the model's outputs deserve to know that the system was evaluated carefully on representative inputs, that failure modes were documented, and that performance claims are grounded in evidence rather than marketing.

This accountability pressure also explains why evaluation methodology is increasingly scrutinized. When a lab claims "our model achieves state-of-the-art performance on 12 benchmarks," the scientific community and practitioners need to assess whether those benchmarks are relevant, whether the evaluation conditions were fair, and whether the claims would hold under different conditions. Transparency in evaluation methodology is a prerequisite for any meaningful accountability.

Before designing an evaluation, you need clarity on what capability or behavior you are measuring. This seems obvious but is surprisingly easy to conflate.

A model might be capable of producing a correct answer to a question without tending to do so reliably. A model capable of multi-step reasoning might still default to shallow pattern matching on many inputs. The distinction matters because evaluation is usually sampling-based: you draw a finite number of test examples, and whether those examples trigger the capability or just the tendency depends heavily on prompt design, sampling temperature, and the examples chosen.

Evaluation that focuses on capability asks: can this model produce the right answer at all, under any prompting strategy? Evaluation that focuses on tendency asks: does this model reliably produce the right answer in natural conditions, without special prompting? Both are valid but measure different things.

The gap between capability and tendency has grown more important as chain-of-thought prompting and few-shot demonstrations have shown that surface-level performance on benchmarks can change dramatically with prompt design. A model evaluated with zero-shot prompts may score 50% on a reasoning task, while the same model with chain-of-thought prompting scores 75%. Which number reflects "the model's capability"? In one sense, the higher number reflects what the model can do. In another sense, the lower number reflects how it would perform if deployed in a context where users do not write expert prompts. Both numbers are true; they are answering different questions.

Intrinsic metrics measure properties of the model's output directly, without reference to downstream tasks. Perplexity is the canonical intrinsic metric: it measures how well the model predicts held-out text, which is an intrinsic property of the model's learned probability distribution. Other intrinsic metrics include fluency scores, coherence measures, and factual consistency scores.

Extrinsic metrics measure how well the model performs on real downstream tasks. Accuracy on a question-answering benchmark is an extrinsic metric. Code execution success rate is an extrinsic metric. Task completion rate in an agent evaluation is an extrinsic metric. Extrinsic metrics directly measure what we care about but require building task-specific evaluation infrastructure.

In practice, intrinsic metrics are most useful during training as development signals, while extrinsic metrics provide the ground truth about whether a model is useful. As we covered in Perplexity Evaluation from Part LIII, perplexity is a powerful intrinsic signal for language model training, but its correlation with downstream task performance weakens as models scale and as tasks shift from text completion to instruction following.

The weakening correlation between perplexity and task performance is not a surprise when you think about what each measures. Perplexity rewards accurate probability mass on held-out tokens. Task performance rewards the right outputs for specific inputs. A model that assigns a slightly lower probability to the next token in a news article may or may not be better at answering questions about that article. The two objectives overlap in the sense that both require understanding language, but they diverge in ways that become more apparent as models get larger and tasks become more structured.

Most benchmark evaluation is static: you present the model with a fixed prompt and score the response. But real language model deployment is often interactive, involving multi-turn conversations, tool use, or iterative refinement. Static evaluation may not capture these interactive dynamics.

Interactive evaluation is harder to scale but increasingly important as models are deployed in agentic settings. The evaluation methodology must match the deployment scenario, or you risk optimizing for benchmark performance while leaving deployment performance unaddressed.

Static evaluation has an important advantage: it is reproducible. You can run the same prompt on different model versions and get scores that are directly comparable. Interactive evaluation introduces additional variability because the sequence of turns depends on the model's previous outputs, and different model versions will diverge in different ways across a conversation. This makes it harder to attribute score differences to specific model properties.

The practical resolution is to use static evaluation for fast iteration during development and to supplement it with interactive evaluation at major milestones. For production systems, shadow evaluations that replay real user conversations against new model versions combine the realism of interactive evaluation with the control of a fixed evaluation set.

A subtler distinction is between behavioral evaluation and capability evaluation. Behavioral evaluation asks what the model does under realistic conditions: given a realistic distribution of inputs, what fraction of responses are helpful, harmless, and honest? Capability evaluation asks what the model can do under the best possible conditions: given carefully designed prompts, does the model possess the underlying knowledge and reasoning ability?

This distinction matters for model development decisions. If a model lacks a capability, you need to invest in data and training that build that capability. If a model has the capability but does not exhibit the behavior reliably, you need to invest in prompting strategies, instruction tuning, or RLHF that align the behavior with the capability. These require different interventions, and conflating them leads to misdiagnosis of what a model needs.

Safety evaluation is almost always behavioral rather than capability-based: you are not testing what harmful things the model could be made to do with elaborate prompts, but rather what harmful things it tends to do under realistic conditions. This is why red-teaming exercises, which specifically probe for capabilities by designing adversarial prompts, are a distinct activity from standard safety evaluation.

A well-designed evaluation study makes defensible claims about model performance. Poorly designed evaluation studies produce numbers that are easy to misinterpret. The design choices that matter most are test set construction, sample size and statistical power, and experimental controls.

The test set is the foundation of any evaluation. Several properties determine its quality.

Distribution match. The test set should reflect the actual distribution of inputs the model will encounter in deployment. A customer service chatbot evaluated on academic question-answering benchmarks may score well but fail in deployment because the evaluation distribution did not match. Constructing a representative test set requires knowledge of the deployment context: who will use the system, what they will ask, and what kinds of failure modes matter most.

Contamination avoidance. Test data must not appear in the model's training data. This sounds obvious but is increasingly difficult to guarantee when training on large web crawls. A model that has memorized benchmark questions and answers from its training data will score artificially high, not because it has learned the underlying capability but because it has learned the specific test instances. This is called benchmark contamination or training data contamination, and it is a growing problem in large language model evaluation.

Contamination can be detected by checking whether benchmark performance significantly exceeds performance on held-out variants of the same tasks, by running membership inference attacks that test whether specific examples are memorized, or by using time-based splitting where test examples were created after the model's training data cutoff.

Diversity and coverage. A test set should cover the range of inputs the model is expected to handle, including edge cases and challenging examples. A test set that is too easy will not differentiate good models from mediocre ones. A test set that is too narrow will only measure one facet of performance.

Label quality. Incorrect or ambiguous labels produce noisy evaluation signals. For tasks with subjective correct answers, inter-annotator agreement should be measured and reported. For tasks with objective correct answers, the labeling process should be carefully reviewed. Poor label quality is a common source of misleading evaluation results.

Label quality issues are often invisible. If a benchmark contains 5% incorrectly labeled examples, a model that achieves 95% accuracy may be correct on every example with a correct label, but the noise in the labels makes it appear to be making errors. Worse, models that perform better than the labeling accuracy cannot be rewarded even if they are correct, because the benchmark treats the (incorrect) label as ground truth. This phenomenon, sometimes called "label ceiling," means that noisy benchmarks systematically underestimate the performance of the best models while masking the true gap between good and mediocre models.

Temporal and demographic coverage. Test sets that are drawn from a narrow time window or from a single demographic can miss systematic failures. A model that handles English-language queries from American users well might fail on queries in other dialects, languages, or cultural contexts. A model evaluated on news articles from 2020 might have learned to perform well on those specific narratives without generalizing to topics that became prominent later. Building representative test sets requires intentional curation to cover diverse populations, time periods, and topic domains, not just random sampling from whatever data is convenient.

A common mistake is drawing strong conclusions from small evaluation samples. If a model achieves 75% accuracy on a 20-example test set versus 72% on a different model, the difference is almost certainly not statistically significant. Even if one model is better, a 20-example sample cannot reliably detect a 3 percentage-point difference.

The sample size required to detect a difference of magnitude $\delta$ with statistical power $1 - \beta$ and significance level $\alpha$ can be approximated for proportion differences using the one-sample proportion formula:

where:

• $n$: the required number of test examples per group (i.e., per model being compared)
• $z_{\alpha/2}$: the critical value from the standard normal distribution for a two-tailed test at significance level $\alpha$; for the conventional $\alpha = 0.05$, this equals 1.96
• $p$: the expected proportion of correct answers, approximated as the average of the two models' expected performance
• $1 - p$: the variance-related term reflecting that proportions near 0.5 have the highest uncertainty and require the largest samples
• $\delta$: the minimum effect size you wish to detect, expressed as a proportion (e.g., $\delta = 0.03$ to detect a 3 percentage-point difference)

The formula shows why evaluation is expensive at high precision: shrinking $\delta$ by half requires four times as many examples, because $\delta$ appears squared in the denominator. For example, to detect a 5 percentage-point difference around 70% baseline accuracy at $\alpha = 0.05$, you need approximately

To detect a 1 percentage-point difference under the same conditions, you would need approximately 8,000 examples. Most benchmark evaluations use hundreds to thousands of test examples for this reason, and results should always be reported with confidence intervals or statistical tests.

The figure below shows how required sample size grows as the effect size you want to detect shrinks, for different baseline performance levels. Notice the rapid growth as effect size drops below 3 percentage points: this is the regime where most real-world model comparisons operate, and where small test sets are most misleading.

Statistical power has direct practical consequences. A team that runs a 100-example evaluation and finds no significant difference between two model versions might conclude that the models are equivalent and ship whichever is cheaper to run. But the test may simply have lacked the power to detect a real difference. This is called a Type II error: failing to detect a true effect. The probability of making a Type II error is $1 - \text{power}$, so at 80% power you are accepting a 20% chance of missing a real difference. For high-stakes deployment decisions, 80% power may not be sufficient.

The practical solution is to plan sample sizes prospectively, before running the evaluation. Decide what effect size is meaningful, what significance level you require, and what power you want. Then calculate the required sample size. This forces you to think carefully about the question before you look at data, which is the right order.

Good evaluation design controls for confounds that might explain performance differences apart from the capability being measured. Common confounds include:

Prompt sensitivity. LLMs can respond very differently to slight variations in prompt wording. A model that performs better on one phrasing might perform worse on an equivalent phrasing. Evaluation results should be averaged over multiple prompt templates, or the specific template used should be clearly documented and justified.

Decoding parameters. Temperature, top-p, top-k, and beam search parameters all affect output quality. Models compared using different decoding strategies cannot be fairly compared. Standardize decoding parameters across compared models.

Model size and compute. A larger model will typically outperform a smaller one on most benchmarks. Comparing model A to model B without controlling for parameter count or training compute conflates architectural quality with scale. Fair comparisons require matched scale or explicit reporting of scale differences.

Format compliance. Benchmarks that require specific output formats (multiple choice letter, JSON output, specific code syntax) will systematically underestimate performance if a model fails to comply with the format even when it knows the correct answer. Parsing and format compliance should be handled robustly, and format failures should be analyzed separately.

Position effects. Multiple-choice benchmarks are subject to position bias: models may systematically prefer the first option, the last option, or the longest option regardless of content. This bias should be measured by permuting answer choices and checking whether model performance changes. Any benchmark design that does not account for position effects is producing partially confounded measurements.

Demonstration effects in few-shot evaluation. When evaluation uses few-shot prompts with example demonstrations, the choice of examples can substantially affect performance. Two researchers using different sets of few-shot examples for the same model on the same benchmark can arrive at scores that differ by several percentage points. Best practice is either to use zero-shot evaluation or to average over multiple randomly sampled sets of demonstrations.

Evaluation metrics convert model outputs into numerical scores. The choice of metric is as important as the choice of test set, because the metric determines what "better" means in your evaluation. There are several broad families of metrics, each with distinct strengths and weaknesses.

Reference-based metrics compare model outputs against gold-standard reference answers. They are fast, cheap, and reproducible, making them standard for large-scale evaluation during development.

The simplest reference-based metrics are exact match and token-level overlap. Exact match is appropriate when there is a single correct answer (factual questions, code that must have a specific output), and it gives full credit for correct answers and zero credit for everything else. Token overlap metrics like BLEU and ROUGE measure partial credit by comparing n-gram sequences between model output and reference. As we will examine in detail in the Automatic Metrics chapter, these metrics correlate reasonably with human judgments for tasks like machine translation and summarization, but they have well-known failure modes: they reward surface-level similarity over semantic correctness and penalize valid paraphrases.

Semantic similarity metrics like BERTScore address the surface-level limitation by comparing representations in embedding space rather than token sequences. A model output that is semantically equivalent but lexically different from the reference will receive a higher BERTScore than it would from BLEU or ROUGE.

The brevity penalty in BLEU deserves particular attention. Without it, you could achieve 100% precision trivially by generating a single word that appears in the reference. The brevity penalty closes this loophole by requiring that generated text have approximately the same length as the reference. But this introduces a new problem: it assumes the reference is the right length, which is not always true. A model that generates a concise but complete summary will be penalized relative to one that generates a verbose summary that pads out to reference length, even if the concise summary is better.

These limitations are fundamental rather than accidental. N-gram overlap metrics were designed for specific tasks with well-defined references, and they work reasonably well there. When applied outside those settings, their limitations become dominant. The research community has developed a variety of alternatives, from BERTScore and BLEURT (which use contextual embeddings) to task-specific metrics (which measure what matters for the task), but automatic metrics remain an approximation. They are valuable for the speed and scale they enable, not as ground truth about quality.

Many evaluation tasks have established, purpose-built metrics. Some important examples:

Accuracy and F1 for classification tasks. Accuracy measures the fraction of examples classified correctly. F1 balances precision and recall, which matters when class distributions are imbalanced or when false positives and false negatives have different costs.

Exact match and character error rate for extraction and transcription tasks. These measure whether the model extracts precisely the right span of text from a document, or transcribes speech with minimal character-level error.

Execution success rate and test pass rate for code generation. A code model is most naturally evaluated by whether the code it generates runs and produces correct outputs, not by how similar the generated code looks to a reference solution.

Factual accuracy and citation correctness for retrieval-augmented generation. These require external verification against a knowledge base or documents, which adds infrastructure cost but provides more meaningful signals than surface-level comparisons.

Task-specific metrics are more meaningful than general automatic metrics precisely because they measure the right thing for the specific task. Code execution rate is a far better measure of coding ability than BLEU similarity to a reference solution, because two code snippets can be functionally identical while being textually very different, and BLEU would penalize this. Likewise, for structured output tasks like slot filling, exact match on the extracted fields is far more informative than token overlap.

The cost of task-specific metrics is that they require custom infrastructure. You need a code execution environment with the right language versions, libraries, and sandboxing. You need a knowledge base and fact-checking pipeline. You need structured output parsers that can handle format variations. This infrastructure takes time to build and maintain, which is why teams often start with general automatic metrics and graduate to task-specific metrics as they scale.

Human evaluation asks people to judge model outputs directly, using scales like Likert ratings, preference comparisons, or detailed rubrics. It is the most direct way to measure subjective qualities like helpfulness, fluency, coherence, and appropriateness, which automatic metrics struggle to capture.

Human evaluation is expensive and time-consuming. A single evaluation study involving hundreds of examples and multiple annotators can cost thousands of dollars and take weeks. This limits how frequently it can be used during development. It is best reserved for major milestones: comparing production candidates, measuring alignment properties, or validating that automatic metrics are tracking human judgments correctly.

Inter-annotator agreement, measured using Cohen's kappa or Krippendorff's alpha, is a critical diagnostic for human evaluation. Low agreement means annotators are not measuring the same thing, which makes the results unreliable. Agreement should always be reported alongside human evaluation results.

Human evaluation is also subject to its own biases that are easy to overlook. Annotators are influenced by the order in which responses are presented. They apply different standards depending on the context and their prior experience. They may agree with well-written prose even when it is factually wrong, because fluency and accuracy are not the same thing. Designing annotation protocols that are clear, detailed, and tested with pilot studies before the main evaluation is essential for getting reliable results.

The most common failure mode in human evaluation is giving annotators a task that is underspecified. "Rate this response on a scale of 1 to 5 for helpfulness" seems clear, but different annotators will apply different implicit criteria. One annotator might focus on whether the response is factually accurate. Another might focus on whether it directly addresses the question. A third might weight tone and politeness. Without a detailed rubric that specifies what each score level means and what dimensions to consider, annotators will diverge, agreement will be low, and the evaluation will produce noisy results. Invest time in rubric design before running any human evaluation.

A newer evaluation paradigm uses strong language models to rate the outputs of weaker or different models. GPT-4 evaluating GPT-3.5 outputs, for example, or using a dedicated judge model to score responses on a rubric. This approach is faster and cheaper than human evaluation while correlating reasonably with human judgments on many dimensions.

The appeal of LLM-as-judge is that language models can assess properties like helpfulness, coherence, and instruction-following that are difficult to measure with n-gram overlap but easy for a capable model to recognize. A model can understand that two responses convey the same information even when they use different words, and it can assess whether a response follows the spirit of a request rather than just its literal wording. These are exactly the qualities where automatic metrics fail and human evaluation is expensive.

LLM-as-judge has notable failure modes. Strong position bias causes judge models to systematically prefer the first answer when two options are presented, even when the two answers are randomly swapped. Verbosity bias causes judge models to prefer longer answers regardless of quality, rewarding quantity over correctness. Self-enhancement bias causes a judge model to prefer outputs that resemble its own generation style, creating systematic favoritism when the judge and one of the evaluated models share architecture or training data.

These biases must be mitigated through careful evaluation design. Presenting each pair in both orders and averaging the scores addresses position bias. Explicitly instructing the judge to value conciseness and checking that brevity is not penalized addresses verbosity bias. Using human-written reference answers or multiple diverse judge models addresses self-enhancement bias. We will examine this evaluation paradigm in depth in the LLM-as-Judge chapter, including experimental evidence on how large these biases are and which mitigation strategies work best.

Despite its limitations, LLM-as-judge has become a practical workhorse for model evaluation, particularly for instruction-following and open-ended generation tasks where no good automatic metric exists. Systems like MT-Bench and AlpacaEval use LLM-as-judge as their primary evaluation mechanism, and this has enabled rapid iteration on model quality during post-training at a fraction of the cost of human evaluation. The key is to treat LLM-as-judge scores as estimates with known biases that require calibration, not as ground truth.

Benchmark suites like MMLU, HellaSwag, and ARC bundle hundreds or thousands of multiple-choice questions across diverse domains into standardized evaluation protocols. They enable rapid comparison across many models and have become the standard way to track progress in the research community.

The strength of benchmark suites is their breadth and standardization. The weakness is that they can become outdated as models improve, and performance improvements on a benchmark may not reflect real capability improvements if the benchmark has been contaminated or if the model has been specifically tuned to perform well on it. We will examine major benchmark suites and their design properties in the Benchmark Design chapter.

The figure below positions the main evaluation approaches across two key dimensions: evaluation cost and alignment with human judgment. No single approach dominates; the choice depends on which tradeoff matters most for your current development stage.

Evaluation is not a one-time activity but a continuous process that evolves with the model development cycle. Understanding how evaluation fits into each stage of development helps you apply the right evaluation method at the right time.

During active model development, the primary evaluation goal is fast feedback. You are making many changes, and you need to know quickly whether each change improves or degrades performance. This requires cheap, automated evaluation that can run in minutes or hours rather than days.

Perplexity on a held-out validation set is the fastest signal: it is computed as a byproduct of training and requires no additional inference. For instruction-tuned models, a small set of representative prompts with automatic scoring, whether by regex matching, execution, or simple LLM-as-judge calls, provides the next layer of signal without the cost of a full benchmark evaluation.

The key discipline in this phase is not to over-index on any single signal. Improving validation perplexity is a good sign, but it does not guarantee improvement on instruction-following tasks. A small representative evaluation suite that covers the main capability areas you care about, run after each significant training change, provides far more actionable signal than perplexity alone.

Once training runs are complete and you have candidate model checkpoints, the evaluation focus shifts to comparison and selection. This is the stage where you need evaluation that is both broad and statistically reliable.

Best practice is to maintain a fixed, versioned evaluation suite for this purpose. Running all candidate checkpoints against exactly the same evaluation suite, in exactly the same conditions, ensures that score differences reflect model differences rather than evaluation variability. Any change to the evaluation suite, including updates to prompts or test sets, should be versioned and tracked so that comparisons across time remain valid.

For major checkpoints, supplement automatic evaluation with targeted human evaluation on a sample of outputs. Human evaluation is expensive, so focus it on the dimensions where automatic metrics are least reliable: instruction-following fidelity, response style, tone, and subjective quality. Human evaluation at this stage is used to validate that the automatic metrics are tracking what you care about, and to catch regressions that the automatic suite might miss.

Before deploying a model to users, evaluation must expand to cover dimensions that are irrelevant during training but critical in production: safety, reliability under adversarial inputs, behavior at the edges of the intended use case, and consistency across multiple runs.

Safety evaluation requires red-teaming: deliberately attempting to elicit harmful outputs through adversarial prompts, jailbreaks, and edge cases. This is a specialized form of evaluation that goes beyond standard benchmarks and requires domain expertise in the safety risks relevant to the deployment context. A customer service chatbot requires different safety evaluation than a medical information assistant, because the harmful outputs and the adversarial techniques differ.

Reliability evaluation checks whether the model behaves consistently under small variations in prompt wording, order of information, or irrelevant context. A model that gives a correct answer to a question stated one way but an incorrect answer to the same question stated differently is less reliable than its benchmark scores suggest. Measuring this variance requires prompt perturbation experiments that go beyond standard evaluation.

After deployment, evaluation must track how the model performs in the wild, on the actual distribution of user inputs rather than the evaluation distribution. This requires logging a representative sample of production inputs and outputs, and applying evaluation metrics to this production sample on a regular cadence.

Production evaluation reveals distribution shift: the gradual divergence between the deployment distribution and the evaluation distribution. It also reveals new failure modes that were not anticipated during pre-deployment evaluation. A model deployed to a diverse user base will encounter prompts that no evaluation designer imagined, and some of those prompts will expose unexpected failures.

The feedback loop from production evaluation back to model development is one of the most valuable signals in the entire pipeline. Teams that invest in production evaluation infrastructure and use the signal to inform training data curation and fine-tuning decisions improve their models faster than teams that rely solely on benchmark evaluation.

Understanding where evaluation goes wrong is as important as understanding how to design evaluation correctly. Several failure modes recur across language model evaluations, and recognizing them is essential for interpreting published results critically.

Benchmark contamination occurs when questions or answers from a benchmark appear in a model's training data. Because modern language models are trained on enormous web crawls that include academic papers, GitHub repositories, and public datasets, contamination is difficult to avoid entirely. A model that has seen MMLU questions in its training data will score higher on MMLU not because it is more capable, but because it is recalling memorized answers.

Contamination detection is an active area of research. Simple heuristics check for exact or near-exact string matches between benchmark items and training data. More sophisticated methods use perplexity probes: if a model assigns unexpectedly low perplexity to benchmark items relative to semantically similar but uncontaminated text, this is evidence of memorization. Time-based evaluation, where benchmarks are constructed from events or publications after a known training cutoff date, provides the strongest contamination protection but limits the topics that can be evaluated.

The contamination problem is most severe for models that train on the latest snapshot of the web, because benchmark items that were created years ago are well-represented in web crawls. This creates a troubling dynamic: the most capable and widely deployed models are also the most likely to be contaminated on established benchmarks, making it hard to interpret their scores at face value.

Benchmark saturation occurs when models improve so much on a benchmark that performance approaches the ceiling, leaving no room to distinguish between good and excellent models. MMLU reached near-human performance thresholds for frontier models around 2023-2024, meaning that small differences in reported scores no longer carried meaningful signal about capability differences. The benchmark had outlived its usefulness for comparing frontier models, though it remained informative for evaluating smaller or less capable models.

The response to saturation is to develop harder benchmarks and to retire old ones as they saturate. This is an ongoing process: today's challenging benchmark becomes tomorrow's easy one. The community now maintains progressively harder benchmarks for specific capabilities, and some evaluation frameworks explicitly track which benchmarks are "solved" and remove them from aggregate scores to preserve discrimination.

Even without explicit contamination, evaluation can be gamed. Models that are fine-tuned on broad instruction-following datasets that happen to include benchmark-style questions will learn patterns that transfer to benchmark performance without transferring to real-world usefulness. This is sometimes called "benchmark overfitting" and it is structurally similar to the reward hacking problem in RLHF: the model learns to maximize the measured signal without necessarily improving the underlying capability the signal was meant to proxy.

The mitigation is to maintain evaluation sets that are unknown to the training team, to diversify evaluation formats so that no single format can be targeted by training, and to continuously update evaluation sets rather than treating them as fixed.

Metric-goal misalignment occurs when the metric used in evaluation does not reflect the actual goal of the system. A summarization model optimized for ROUGE-L may learn to produce summaries that are lexically similar to reference summaries but miss the key information that a human reader would want. A question-answering model optimized for exact match may fail on questions where the correct answer can be expressed in multiple equivalent forms.

This failure mode is pervasive and difficult to eliminate because it is structural: any metric you can compute is a simplification of the goal. The best defense is to use multiple metrics that make different simplifications, so that a model that games one metric will be caught by another, and to periodically validate automatic metric scores against human judgments to ensure the proxy relationship is holding.

Good evaluation practice requires attention to both methodological rigor and practical constraints. The most important principles are detailed below.

No single metric captures the full picture of model quality. Use a combination of:

• Automatic metrics for fast iteration during development
• Benchmark suites for standardized comparison with other models
• Human evaluation for high-stakes quality assessments
• Task-specific metrics tailored to your deployment use case

Convergence across multiple evaluation approaches increases confidence in conclusions. Divergence signals that different approaches are measuring different things, which is worth understanding. When automatic metrics and human judgments disagree, the disagreement is often informative: it reveals that the automatic metric has a blind spot that matters for the specific task or capability being evaluated.

Point estimates of benchmark performance without confidence intervals are difficult to interpret. Small differences in reported numbers may not be statistically meaningful. Always report confidence intervals, standard deviations across multiple evaluation runs, or results of statistical significance tests when comparing models.

Reporting practices in the research community have historically been poor in this regard. Many benchmark comparisons treat 1-2 point differences as meaningful when they are well within the margin of sampling variability. The broader adoption of statistical reporting would improve the quality of conclusions drawn from evaluation studies, and practitioners reading published results should always ask: how large is the standard error here, and would this difference hold on a different test set?

Evaluation results are only reproducible if all conditions are documented:

• The exact prompt templates used for each task
• Decoding parameters (temperature, top-p, max tokens)
• Model checkpoint and version
• Test set version and any filtering applied
• Scoring code and tie-breaking rules

Undocumented evaluation conditions are a major reason why published results cannot be reproduced and why different teams report different scores for the same model on the same benchmark. When a research paper reports a new state-of-the-art result, the evaluation conditions are often as important as the model architecture for explaining the improvement.

Goodhart's Law states that when a measure becomes a target, it ceases to be a good measure. In LLM evaluation, this manifests as models being fine-tuned specifically on benchmark data (even without explicit contamination) or as evaluation pipelines being designed to produce favorable numbers rather than accurate measurements.

Building evaluation suites that are difficult to game requires continuous refresh of test data, diversity of evaluation formats that cannot all be simultaneously optimized, and honest acknowledgment that benchmark performance is a proxy, not the goal.

A practical corollary is to maintain held-out evaluation sets that the training team does not know about. If the development team never sees the test set, they cannot optimize for it, whether intentionally or inadvertently through choices about training data curation and instruction tuning design.

The most commonly violated principle of evaluation is the simplest: evaluate on the distribution that matters. A coding assistant evaluated on mathematical reasoning benchmarks may show impressive numbers while failing on its actual use case. Construct evaluation sets that represent real queries from real users whenever possible. Usage logs from deployed systems, or user studies with realistic tasks, provide far more relevant signal than academic benchmarks for production systems.

As a practical workflow, teams building production systems should maintain two separate evaluation sets: a public benchmark suite for comparison with the research community and for tracking broad capability trends, and a private deployment evaluation set built from real usage data for making production decisions. The public benchmarks provide context and enable external validation. The private deployment set provides the signal that matters for the product.

Let's walk through how to apply these principles to design an evaluation for a model intended to summarize news articles.

Step 1: Define the evaluation goals. We want to measure (a) factual accuracy, (b) coverage of key points, (c) conciseness, and (d) fluency. These correspond to different failure modes: a model might be fluent but factually wrong, or accurate but verbose, or concise but missing key information. Each of these dimensions must be measured separately, because they are not correlated and a model that excels on one may fail on another.

Step 2: Choose metrics for each goal. Factual accuracy requires human annotation or an automated fact-checking system; ROUGE scores will not catch factual errors because they measure token overlap rather than semantic truth. Coverage can be approximated by ROUGE recall (does the summary contain the key n-grams from the article?). Conciseness can be measured by compression ratio. Fluency can be measured by a language model perplexity score or by human raters.

Step 3: Construct the test set. We collect 500 news articles from the target deployment domain (not from CNN/DailyMail if we are not deploying on CNN/DailyMail). We verify these articles are not in the model's training data by checking against dates and source identifiers. We collect reference summaries from professional editors.

Step 4: Plan statistical analysis. With 500 examples, we can detect differences of approximately 2 percentage points in ROUGE-1 recall at 95% confidence. We document this power calculation before running the evaluation, committing to the sample size before seeing results.

Step 5: Establish controls. All models are evaluated with the same prompt template, the same decoding temperature, and the same maximum output length. We document these choices explicitly. When comparing multiple models, we randomize evaluation order to avoid any systematic effects.

Step 6: Conduct human evaluation on a sample. We cannot afford human evaluation on all 500 examples, so we randomly sample 50 and have three annotators rate factual accuracy and coverage on a 5-point Likert scale. We compute inter-annotator agreement before proceeding. If agreement is too low (kappa below 0.4), we revise the annotation guidelines and re-annotate.

Step 7: Interpret results in context. A 3-point improvement in ROUGE-1 that does not correspond to improvement in human factual accuracy ratings suggests the model is adding words that match reference summaries without being more accurate. We report both metrics and flag the discrepancy.

This kind of multi-metric, principled evaluation design is what distinguishes useful evaluation from evaluation theater.

Let's implement the core components of an evaluation pipeline. We will compute several metrics for a summarization task, demonstrate significance testing, and show how to build a simple evaluation report.

Model A outputs are more specific and factually complete, while Model B outputs are vague and lose key quantitative details. Let's measure this difference with metrics.

ROUGE measures n-gram overlap between a generated summary and one or more reference summaries. ROUGE-1 measures unigram (single token) overlap. ROUGE-2 measures bigram overlap. ROUGE-L measures the longest common subsequence. ROUGE recall asks how much of the reference content appears in the hypothesis, and this is typically the most informative variant for summarization because the primary failure mode is leaving important information out.

Model A shows higher ROUGE recall because it retains more specific details from the reference. ROUGE-2 differences are larger than ROUGE-1 because exact phrase matches require both the tokens and their order to be correct, which rewards Model A's more faithful reproduction of key phrases.

For tasks where there is a definitive correct answer, exact match is the right metric. Let's also compute a simple coverage metric that counts how many key entities from the reference appear in the summary.

A difference in mean scores only matters if it is statistically significant. With a small test set, apparent differences can be due to sampling variability. We use the bootstrap resampling method to compute confidence intervals.

Bootstrap resampling works by repeatedly drawing samples of the same size as the original test set, with replacement, and computing the metric difference on each resample. The distribution of these bootstrap differences approximates the sampling distribution of the true difference, and the percentiles of this distribution give us confidence interval bounds. Bootstrap is preferred over parametric tests here because it makes no assumptions about the distribution of metric values.

With only 5 examples, neither difference reaches statistical significance at 95% confidence. This illustrates a critical point: you cannot draw strong conclusions from small evaluation samples, even when the mean differences look meaningful. The confidence intervals span nearly the full range of possible outcomes. The right response to this result is not to conclude the models are equivalent, but to recognize that the test is underpowered and more examples are needed before drawing any conclusion.

Let's implement the sample size formula from earlier to understand how many examples we need for reliable evaluations.

Detecting small differences requires large test sets. A 1 percentage-point difference at 70% baseline accuracy requires over 3,000 examples. This is why well-designed benchmark suites use hundreds to thousands of test examples: smaller sets simply cannot reliably detect meaningful performance differences.

Understanding the bootstrap distribution helps build intuition about why small samples produce wide confidence intervals. Let's visualize the bootstrap distribution for our ROUGE-1 difference.

A structured evaluation report brings all metrics together and documents the conditions under which they were measured.

The evaluation report makes it easy to communicate results in a structured way and documents the conditions that allow results to be reproduced or challenged.

The core parameters for any evaluation pipeline are:

• n_bootstrap: Number of resampling iterations for bootstrap confidence intervals. Higher values (2000 or more) produce stable interval estimates.
• confidence: The desired confidence level for intervals, typically 0.95 for 95% confidence.
• alpha (significance level): Probability of a false positive. Standard choice is 0.05, meaning a 5% chance of concluding a difference is real when it is not.
• power: Probability of detecting a true difference. 0.80 is the conventional minimum; 0.90 is better for high-stakes comparisons.
• delta: The minimum effect size worth detecting, expressed as a proportion. Setting this too small (e.g., 0.005) requires impractically large test sets.

Evaluation of language models is fundamentally harder than evaluation of traditional software systems. A rule-based system that either passes or fails test cases is much easier to evaluate than a generative system that produces open-ended outputs with many possible forms of correctness.

The most pervasive limitation is the proxy problem: every metric is a proxy for what we care about, and all proxies break down at the edges. ROUGE does not capture factual accuracy. Benchmark accuracy does not capture real-world helpfulness. Human preference ratings do not capture long-term safety properties. This does not mean metrics are useless: they provide real signal about real properties. But it does mean you should maintain healthy skepticism about conclusions drawn from any single metric.

Benchmark saturation is an increasingly pressing problem. As models improve, they approach and sometimes exceed human performance on established benchmarks like MMLU and HellaSwag. At this point, the benchmark no longer discriminates between good and excellent models, and progress appears to stall even when real capability improvements are occurring. The research community must continuously develop harder benchmarks that remain informative as models improve.

Contamination threatens to invalidate benchmark results for the most capable models. When training data covers much of the internet and many academic papers, it inevitably includes benchmark questions and answers. Carefully designed future benchmarks will need to be generated after a known date and verified not to appear in training corpora.

Distribution shift between evaluation and deployment is often underestimated. Users interact with language models in ways that evaluation designers did not anticipate. Tasks that were not represented in the evaluation suite become common in deployment. The best mitigation is continuous evaluation using real usage logs from deployed systems, which provides direct feedback about deployment distribution.

The gap between academic evaluation and production evaluation is substantial enough that some organizations maintain entirely separate teams for each. Academic evaluation tracks progress against the research community and provides the external signal needed for scientific credibility. Production evaluation tracks deployment quality and provides the internal signal needed for engineering decisions. The best organizations treat these as complementary rather than equivalent, using academic benchmarks for broad capability tracking and production metrics for product decisions.

Despite these limitations, good evaluation practice remains far better than no evaluation or naive evaluation. Rigorous evaluation reveals failure modes before deployment, tracks progress accurately, and enables principled comparison between model versions and architectures. The chapters that follow will equip you with the specific tools to apply these principles across different evaluation scenarios.

Evaluation is the systematic measurement of language model capabilities and behaviors. In this chapter, we covered the foundational concepts you need to design and interpret evaluations correctly.

The core evaluation concepts:

• Evaluation goals: tracking progress, selecting models, communicating capabilities, and validating that training objectives generalize to real-world use
• Capability vs. tendency: a model may be capable of something without reliably tending to do it; evaluation design determines which you are measuring
• Intrinsic vs. extrinsic metrics: intrinsic metrics measure model properties in isolation; extrinsic metrics measure task performance that represents real use cases
• Static vs. interactive evaluation: static evaluations present fixed prompts; interactive evaluations measure multi-turn dynamics that matter in agentic deployment
• Behavioral vs. capability evaluation: behavioral evaluation measures what a model does under realistic conditions; capability evaluation measures what it can do under ideal conditions

The key design principles:

• Test set construction requires distribution match, contamination avoidance, diversity, label quality, and temporal and demographic coverage
• Sample size and statistical power must be planned before running evaluations; small samples cannot reliably detect meaningful differences
• Controls for confounds, including prompt sensitivity, decoding parameters, position effects, and model scale, are essential for fair comparisons

The main metric families:

• Automatic reference-based metrics (BLEU, ROUGE, BERTScore) are fast and reproducible but are proxies for quality
• Task-specific metrics (accuracy, F1, execution success rate) measure what directly matters for specific tasks
• Human evaluation is the gold standard for subjective qualities but is expensive and requires inter-annotator agreement measurement
• LLM-as-judge offers a scalable middle ground between automatic metrics and human evaluation, with known biases that must be managed
• Benchmark suites enable standardized comparison but are subject to saturation and contamination

Common failure modes to watch for:

• Benchmark contamination: test data appearing in training data inflates scores without reflecting underlying capability
• Benchmark saturation: ceilings on established benchmarks mask real capability differences between frontier models
• Overfitting to evaluation: training choices that target benchmark performance without improving underlying capability
• Metric-goal misalignment: the metric measures something related to but not identical to what the system is supposed to do

The next chapters in this part build on these foundations, examining each evaluation paradigm in detail. In Automatic Metrics, we will derive the BLEU, ROUGE, and BERTScore formulas from first principles and understand their failure modes precisely. In Human Evaluation, we will cover annotation study design, inter-annotator agreement, and how to get reliable signal from human raters. In LLM-as-Judge, we will examine the best practices for using strong models as evaluators and how to mitigate systematic biases.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about evaluation fundamentals for language models.