Evaluation Fundamentals › What We Are Measuring

Before designing an evaluation, you need clarity on what capability or behavior you are measuring. This seems obvious but is surprisingly easy to conflate.

A model might be capable of producing a correct answer to a question without tending to do so reliably. A model capable of multi-step reasoning might still default to shallow pattern matching on many inputs. The distinction matters because evaluation is usually sampling-based: you draw a finite number of test examples, and whether those examples trigger the capability or just the tendency depends heavily on prompt design, sampling temperature, and the examples chosen.

Evaluation that focuses on capability asks: can this model produce the right answer at all, under any prompting strategy? Evaluation that focuses on tendency asks: does this model reliably produce the right answer in natural conditions, without special prompting? Both are valid but measure different things.

The gap between capability and tendency has grown more important as chain-of-thought prompting and few-shot demonstrations have shown that surface-level performance on benchmarks can change dramatically with prompt design. A model evaluated with zero-shot prompts may score 50% on a reasoning task, while the same model with chain-of-thought prompting scores 75%. Which number reflects "the model's capability"? In one sense, the higher number reflects what the model can do. In another sense, the lower number reflects how it would perform if deployed in a context where users do not write expert prompts. Both numbers are true; they are answering different questions.

Intrinsic metrics measure properties of the model's output directly, without reference to downstream tasks. Perplexity is the canonical intrinsic metric: it measures how well the model predicts held-out text, which is an intrinsic property of the model's learned probability distribution. Other intrinsic metrics include fluency scores, coherence measures, and factual consistency scores.

Extrinsic metrics measure how well the model performs on real downstream tasks. Accuracy on a question-answering benchmark is an extrinsic metric. Code execution success rate is an extrinsic metric. Task completion rate in an agent evaluation is an extrinsic metric. Extrinsic metrics directly measure what we care about but require building task-specific evaluation infrastructure.

In practice, intrinsic metrics are most useful during training as development signals, while extrinsic metrics provide the ground truth about whether a model is useful. As we covered in Perplexity Evaluation from Part LIII, perplexity is a powerful intrinsic signal for language model training, but its correlation with downstream task performance weakens as models scale and as tasks shift from text completion to instruction following.

The weakening correlation between perplexity and task performance is not a surprise when you think about what each measures. Perplexity rewards accurate probability mass on held-out tokens. Task performance rewards the right outputs for specific inputs. A model that assigns a slightly lower probability to the next token in a news article may or may not be better at answering questions about that article. The two objectives overlap in the sense that both require understanding language, but they diverge in ways that become more apparent as models get larger and tasks become more structured.

Most benchmark evaluation is static: you present the model with a fixed prompt and score the response. But real language model deployment is often interactive, involving multi-turn conversations, tool use, or iterative refinement. Static evaluation may not capture these interactive dynamics.

Interactive evaluation is harder to scale but increasingly important as models are deployed in agentic settings. The evaluation methodology must match the deployment scenario, or you risk optimizing for benchmark performance while leaving deployment performance unaddressed.

Static evaluation has an important advantage: it is reproducible. You can run the same prompt on different model versions and get scores that are directly comparable. Interactive evaluation introduces additional variability because the sequence of turns depends on the model's previous outputs, and different model versions will diverge in different ways across a conversation. This makes it harder to attribute score differences to specific model properties.

The practical resolution is to use static evaluation for fast iteration during development and to supplement it with interactive evaluation at major milestones. For production systems, shadow evaluations that replay real user conversations against new model versions combine the realism of interactive evaluation with the control of a fixed evaluation set.

A subtler distinction is between behavioral evaluation and capability evaluation. Behavioral evaluation asks what the model does under realistic conditions: given a realistic distribution of inputs, what fraction of responses are helpful, harmless, and honest? Capability evaluation asks what the model can do under the best possible conditions: given carefully designed prompts, does the model possess the underlying knowledge and reasoning ability?

This distinction matters for model development decisions. If a model lacks a capability, you need to invest in data and training that build that capability. If a model has the capability but does not exhibit the behavior reliably, you need to invest in prompting strategies, instruction tuning, or RLHF that align the behavior with the capability. These require different interventions, and conflating them leads to misdiagnosis of what a model needs.

Safety evaluation is almost always behavioral rather than capability-based: you are not testing what harmful things the model could be made to do with elaborate prompts, but rather what harmful things it tends to do under realistic conditions. This is why red-teaming exercises, which specifically probe for capabilities by designing adversarial prompts, are a distinct activity from standard safety evaluation.