Agent Evaluation: Metrics, Benchmarks and Safety Standards › Task Completion Metrics

The most intuitive measure of agent performance is whether it accomplishes the assigned goal. This outcome-oriented perspective aligns with how end users typically judge automated systems: did the agent book the flight, fix the bug, or send the report as requested? However, binary success or failure masks important nuances in capability that are needed for diagnosing weaknesses and guiding improvement. We need granular metrics that can distinguish between catastrophic failure, partial success with acceptable compromises, and optimal execution that minimizes resource consumption and maximizes reliability.

The challenge of defining "success" is itself non-trivial and forms the first methodological hurdle in agent evaluation. For a web navigation task, success might mean the agent clicks the correct button to complete a purchase. But what if the agent places the order correctly but in the wrong size? What if it books the right flight but fails to apply a discount code that was mentioned in the task? What if it sends the confirmation email to a slightly wrong address? These gradations of success require evaluation frameworks advanced enough to capture partial credit while remaining consistent and interpretable across thousands of evaluation examples.

At the simplest conceptual level, task completion is binary. An agent either reaches the goal or does not. For a web navigation task, this might mean successfully booking a flight to a specific destination on particular dates. For code generation, it means passing all test cases in the evaluation suite. For data analysis, it means creating a report that contains the correct statistical conclusions.

However, binary metrics prove inadequate for complex tasks with multiple subgoals or intermediate deliverables. Consider an agent asked to "research climate change impacts on agriculture, summarize three findings, and email the report to the team." A purely binary evaluation would treat an agent that retrieves relevant documents but fails to send the email as equally unsuccessful as an agent that retrieves completely irrelevant documents. This loss of signal makes it impossible to track incremental progress during development or to identify which capabilities need strengthening.

To address this limitation, we might define a partial credit scheme that decomposes the task into constituent components:

• 0.3 points for retrieving relevant documents from credible sources
• 0.4 points for extracting three valid, substantiated findings
• 0.3 points for sending the email with correct content and appropriate recipients

This rubric-based approach allows for fine-grained assessment of capabilities, revealing whether an agent struggles with information retrieval, synthesis, or execution. However, this decomposition requires carefully annotating task-specific rubrics for each evaluation domain. A rubric for a web navigation task looks entirely different from a rubric for a software engineering task, and developing consistent, principled rubrics at scale demands significant annotation effort. The rubric approach scales poorly to open-ended domains where task definitions are fluid, but it gives invaluable diagnostic insight into specific failure modes that aggregate metrics would obscure. In research settings, rubric-based decomposition has proven needed for identifying whether model improvements affect planning, execution, or synthesis capabilities independently.

The primary metric for agent evaluation remains the success rate ($\text{SR}$), calculated as the fraction of tasks completed successfully across an evaluation set:

$$\text{SR} = \frac{\text{Number of successful tasks}}{\text{Total number of tasks}}$$

where:

• $\text{SR}$: the success rate, representing the proportion of tasks completed successfully (ranging from 0 to 1)
• $\text{Number of successful tasks}$: count of tasks where the agent achieved the specified goal according to the evaluation criteria
• $\text{Total number of tasks}$: the complete set of evaluation tasks attempted

This ratio gives an intuitive measure of reliability: a value of 0.8 indicates the agent succeeds four times out of five, which is immediately interpretable by practitioners and stakeholders alike. By normalizing by the total task count, we can compare performance across different benchmark sizes and track improvements over time as the agent is refined. Success rate is the headline metric for comparing different agent architectures or model versions, and most published results on major benchmarks report this figure prominently.

When dealing with partial credit schemes like the rubric-based evaluation described earlier, we use normalized task success (NTS) to aggregate performance across tasks with different maximum possible scores:

$$\text{NTS} = \frac{1}{N} \sum_{i=1}^{N} \frac{\text{Points earned}_i}{\text{Points possible}_i}$$

where:

• $N$: total number of tasks evaluated in the dataset
• $\text{Points earned}_i$: the credit received for task $i$, based on completion of specific subgoals or rubric criteria
• $\text{Points possible}_i$: maximum achievable credit for task $i$, which may differ across tasks of varying complexity
• $\sum_{i=1}^{N}$: summation across all tasks from 1 to $N$

This metric extends binary success to capture partial progress, which is needed for complex tasks with multiple subgoals. By normalizing each task's score to the range $[0, 1]$ before averaging, we prevent large, complex tasks from dominating the metric while preserving proportional credit for partial completion. This allows you to detect incremental improvements even when full task completion remains rare, which is necessary in early stages of agent development when zero-to-partial progress is the primary signal of learning.

For tasks with multiple independent subgoals, we calculate subgoal success rate separately from overall task success. This decomposition reveals whether failures stem from high-level planning (missing subgoals entirely) or execution failures (failing at specific steps despite correct planning). An agent with high subgoal success but low overall success may have coordination problems that arise when subgoals must be sequenced correctly or when outputs from one subgoal must feed into another. Low subgoal success across specific skill categories suggests basic capability gaps that require targeted training or architectural changes rather than general fine-tuning.

Code generation evaluation introduced Pass@k, measuring whether any of $k$ independent attempts succeed. This metric extends naturally to agent systems and gives important insights into the reliability and exploration characteristics of agent behavior. An agent with $\text{Pass@}1 = 0.3$ but $\text{Pass@}10 = 0.8$ suggests that the system possesses the underlying capability to solve the task but lacks reliability, possibly due to non-deterministic tool outputs, stochastic exploration behavior, or sensitivity to initial reasoning steps.

This pattern, where performance increases materially with more attempts, indicates that the agent is exploring a solution space rather than deterministically executing a known procedure. While exploration is useful for novel problems, deployed systems require high Pass@1 performance because users typically interact with a single execution thread. The gap between Pass@1 and Pass@k motivates research into self-consistency techniques, where agents generate multiple reasoning paths and select the most common or confident solution. The magnitude of this gap also informs decisions about temperature settings during inference: high-temperature agents explore more aggressively, potentially finding solutions that low-temperature agents miss, but they pay for this exploration with reduced first-attempt reliability.

Consistency measures whether the agent reaches the same outcome given identical initial conditions and task specifications. High variance in success rates across multiple runs of the same task indicates brittleness, which is a necessary concern for deployed systems where reproducible behavior is needed for user trust and system integration. Inconsistency may stem from temperature settings in language model sampling, race conditions in tool execution, or non-deterministic exploration strategies that sometimes find effective paths and sometimes wander into dead ends. A deeply inconsistent agent is difficult to debug, difficult to improve, and difficult to trust, even if its average success rate appears acceptable.

A growing challenge in agent evaluation is that many tasks do not have a single objectively correct answer that can be verified programmatically. When an agent is asked to "write a professional email to reschedule a meeting" or "summarize the key risks in this financial document," success cannot be determined by string matching or code execution. The output must be judged for relevance and completeness as well as tone and accuracy.

LLM judges have emerged as a scalable approach for evaluating these open-ended tasks. A judge model, typically a powerful general-purpose LLM, is given the original task, the agent's full trajectory, and a structured rubric specifying what constitutes a good response. The judge assigns scores and often gives natural language explanations of its assessment. This approach scales better than human evaluation while preserving the fine-grained judgment that binary metrics cannot capture.

However, LLM judges introduce their own reliability challenges. Judges may exhibit self-enhancement bias, where a model of the same family as the evaluated agent gives systematically higher scores to responses in that model's style. They may show length bias, preferring longer responses regardless of quality, or position bias, systematically favoring responses presented first in pairwise comparisons. Calibrating judge behavior requires comparing judge scores against human annotations on a held-out validation set and adjusting prompts or model choices to minimize systematic biases. The meta-evaluation problem, evaluating how well our evaluators evaluate, is a real open problem in agent evaluation research.