Evaluation Fundamentals › Limitations and Practical Implications

Evaluation of language models is fundamentally harder than evaluation of traditional software systems. A rule-based system that either passes or fails test cases is much easier to evaluate than a generative system that produces open-ended outputs with many possible forms of correctness.

The most pervasive limitation is the proxy problem: every metric is a proxy for what we care about, and all proxies break down at the edges. ROUGE does not capture factual accuracy. Benchmark accuracy does not capture real-world helpfulness. Human preference ratings do not capture long-term safety properties. This does not mean metrics are useless: they provide real signal about real properties. But it does mean you should maintain healthy skepticism about conclusions drawn from any single metric.

Benchmark saturation is an increasingly pressing problem. As models improve, they approach and sometimes exceed human performance on established benchmarks like MMLU and HellaSwag. At this point, the benchmark no longer discriminates between good and excellent models, and progress appears to stall even when real capability improvements are occurring. The research community must continuously develop harder benchmarks that remain informative as models improve.

Contamination threatens to invalidate benchmark results for the most capable models. When training data covers much of the internet and many academic papers, it inevitably includes benchmark questions and answers. Carefully designed future benchmarks will need to be generated after a known date and verified not to appear in training corpora.

Distribution shift between evaluation and deployment is often underestimated. Users interact with language models in ways that evaluation designers did not anticipate. Tasks that were not represented in the evaluation suite become common in deployment. The best mitigation is continuous evaluation using real usage logs from deployed systems, which provides direct feedback about deployment distribution.

The gap between academic evaluation and production evaluation is substantial enough that some organizations maintain entirely separate teams for each. Academic evaluation tracks progress against the research community and provides the external signal needed for scientific credibility. Production evaluation tracks deployment quality and provides the internal signal needed for engineering decisions. The best organizations treat these as complementary rather than equivalent, using academic benchmarks for broad capability tracking and production metrics for product decisions.

Despite these limitations, good evaluation practice remains far better than no evaluation or naive evaluation. Rigorous evaluation reveals failure modes before deployment, tracks progress accurately, and enables principled comparison between model versions and architectures. The chapters that follow will equip you with the specific tools to apply these principles across different evaluation scenarios.