Evaluation Fundamentals › Evaluation Fundamentals

You have spent months training a large language model. The loss curves looked good. The model generates fluent text. But is it useful? Does it answer questions accurately, follow instructions reliably, or summarize documents faithfully? You need a way to measure this, and that is what evaluation gives you.

Evaluation is the systematic process of measuring how well a language model performs on tasks that matter. It is not a single activity but a collection of methodologies, each suited to different questions and different stages of development. A model that achieves low perplexity on held-out text might still generate factual errors. A model that scores highly on a multiple-choice benchmark might still struggle with open-ended generation. A model that impresses human annotators in lab settings might behave differently in deployment. Understanding which evaluation approach to apply, when, and how to interpret the results is one of the most practically important skills in working with language models today.

This chapter establishes the conceptual foundations for evaluation: why we evaluate, what we are trying to measure, how evaluation studies are designed, and what the main classes of metrics offer. We will also examine common failure modes in evaluation that can mislead practitioners who rely on them uncritically. The subsequent chapters in this part build on these foundations by examining specific metric families: automatic metrics, human evaluation, LLM-as-judge, and benchmark design.