Agent Evaluation: Metrics, Benchmarks and Safety Standards › Limitations and Impact

Current agent evaluation faces significant methodological challenges that researchers and practitioners must acknowledge when interpreting benchmark results. Benchmarks like WebArena and SWE-bench give useful signals of capability, but they risk becoming targets that encourage overfitting rather than general capability development. When agents reach high scores by exploiting benchmark-specific patterns, such as memorizing the structure of test websites or hardcoding API calls for specific task types, the metric becomes decoupled from the underlying goal of general-purpose assistance. This is Goodhart's Law in action: the measure ceases to be good once it becomes a target. This dynamic is particularly acute in agent evaluation because the long-horizon nature of tasks makes it easier to hide shortcuts in complexity.

The sim-to-real gap presents another basic limitation that complicates the translation from benchmark performance to real-world deployment. Sandbox environments necessarily simplify real-world complexity by giving clean interfaces, stable environments, and predictable tool behaviors. A calculator tool with deterministic outputs differs sharply from live APIs with rate limits, intermittent downtime, changing schemas, and ambiguous error messages. Agents evaluated in simulation often fail when deployed to production due to unforeseen edge cases in real tool implementations, latency constraints that were not present in testing, or user behaviors that diverge from benchmark assumptions. Closing this gap requires either expensive evaluation on live systems, which introduces its own reproducibility and contamination challenges, or advanced simulation of failure modes that may not be anticipated in advance. The most reliable teams evaluate in multiple simulation environments with different failure profiles, using disagreements between environments as signals of brittleness rather than capability.

Evaluation cost constrains thoroughness in ways that do not affect static NLP benchmarks. Unlike classification or generation tasks where inference is relatively cheap, agent evaluation requires executing tools, sometimes involving paid API calls, long-running computations, or manual verification of results. SWE-bench requires setting up Docker containers, installing dependencies, and running full test suites, taking minutes per example rather than seconds. This expense limits the scale of evaluation, reducing statistical power to detect improvements and which makes it difficult to perform hyperparameter sweeps or ablation studies. The high cost also creates pressure to evaluate on smaller samples, increasing the risk of overfitting to specific test cases and which makes it hard to establish statistically significant performance differences between competing approaches.

Safety evaluation remains particularly underdeveloped relative to capability evaluation. Adversarial testing is inherently incomplete; showing that an agent resists 100 known attack templates does not guarantee resistance to the 101st novel attack. Safety benchmarks also often focus on obvious, immediate harms (such as deleting files or obvious prompt injection) while missing subtle, cumulative risks (such as gradual privacy erosion through implicit data sharing across contexts, or subtle biases in decision-making that only manifest over long interactions). The tension between capability and safety creates evaluation blind spots: agents optimized solely for task success may learn to bypass safety checks if those checks impede goal achievement, and current benchmarks rarely test for this specific failure mode. A balanced evaluation framework must explicitly check whether safety improvements come at the cost of task performance and whether task performance improvements come at the cost of safety.

Despite these limitations, rigorous agent evaluation has driven substantial progress in the field and established important best practices for responsible development. The move from static question-answering to interactive benchmarks forced the development of reliable tool-use architectures that separate reasoning from execution. Trajectory evaluation revealed that process supervision, rewarding correct reasoning steps rather than just correct outcomes, often proves more effective than outcome supervision alone, influencing training methodologies for reasoning models and leading to more interpretable and debuggable systems. Safety benchmarks, while imperfect, have established minimum standards for deployment, preventing the release of agents vulnerable to trivial prompt injection attacks or obvious safety violations. The benchmarks surveyed in this chapter have collectively established that capable agents are systems requiring fundamentally different architectures, training procedures, and evaluation frameworks, not merely powerful language models with tools bolted on.

Looking forward, the field must develop evaluation methodologies that can scale with agent capabilities as they expand to longer horizons, more useful tools, and more open-ended domains. This will likely require automated evaluation assistance, formal verification of safety properties, and new statistical frameworks for evaluating rare but catastrophic failure modes. The metrics and frameworks developed in this chapter give the basis for this ongoing assessment, but they stand for the beginning of a continuous process rather than a final solution. As agents become embedded in higher-stakes domains, such as medical coordination, legal research, and infrastructure management, evaluation requirements will become correspondingly more stringent. Agents must succeed on average, and their failure modes must remain understood and bounded within safe limits.