Agent Evaluation: Metrics, Benchmarks and Safety Standards › Agent Evaluation

Evaluating language models that use tools presents a basic challenge that extends far beyond traditional natural language processing benchmarks. Generating coherent, grammatically correct text and factually accurate responses in isolation does not establish success in agent systems. Agent evaluation must capture the ability to reach real-world outcomes through sequences of decisions that unfold over time, interact with external systems, and adapt to changing environmental states. While we explored instruction following evaluation in Part XXXVI, Chapter 6 and retrieval-augmented generation metrics in Part XLIV, Chapter 14, agent evaluation demands entirely new frameworks that capture interaction dynamics, persistence across long horizons, and the tangible impact of actions on external environments.

The move from static evaluation to agent evaluation is a basic change in how we assess artificial intelligence. Traditional benchmarks treat models as functions that map inputs to outputs, where correctness can be verified by comparing the output to a reference answer. Agents, however, are better understood as autonomous systems that maintain internal state, execute actions that modify external state, and engage in extended interactions that may span minutes, hours, or even days. This temporal and causal complexity introduces evaluation dimensions that do not exist in static settings. A question-answering system can be wrong in precisely one way: it gives the wrong answer. An agent can be wrong in dozens of ways: it can plan correctly but execute badly, execute correctly but plan inefficiently, reach the goal but leave behind dangerous side effects, or succeed on benign tasks while failing catastrophically on adversarial ones.

An agent differs from a simple question-answering system in three necessary dimensions that fundamentally change how we measure performance. First, an agent operates over extended trajectories rather than single-turn interactions, taking multiple actions that progressively change the state of the external world. Evaluation must therefore account for the entire interaction history as well as the final output. A trajectory that succeeds via a lucky wrong turn carries very different implications for future reliability than a trajectory that succeeds through principled reasoning. Second, an agent must show the capacity to recover from errors and adapt to dynamic feedback that may be unexpected or contradictory. Unlike static tasks where the input is fully specified at the beginning, agent tasks often involve discovering requirements through interaction, requiring reliable error handling and replanning capabilities. An agent that gives up at the first error has fundamentally different practical utility than one that recognizes failure modes, adapts its approach, and eventually succeeds. Third, an agent's actions carry real consequences that extend beyond digital systems, potentially affecting physical systems, financial transactions, or personal privacy. This requires safety guardrails and behavioral constraints that static benchmarks cannot measure and that traditional accuracy metrics completely ignore.

This chapter develops complete metrics, standardized benchmarks, and safety protocols necessary to evaluate these capabilities responsibly. We begin by examining how to measure task completion in contexts where success is not binary, then explore both the outcomes an agent reaches and the quality of the process used to reach them. We survey current agent benchmarks that give reproducible evaluation environments, and we confront the necessary safety considerations that arise when autonomous systems gain access to tools capable of affecting the real world.