ReAct Pattern: Interleaving Reasoning and Action for LLM Agents › Evaluating ReAct Agents

Evaluating ReAct agents is more complex than evaluating a single forward pass, because there are multiple dimensions of quality to assess. The final answer is the most obvious metric, but it is not the only thing that matters.

The most direct metric is whether the final answer is correct. For factual question-answering benchmarks like HotpotQA (multi-hop questions) and FEVER (fact verification), ReAct was evaluated against pure chain-of-thought baselines. The original Yao et al. paper showed that ReAct improved accuracy by 3-8 percentage points on HotpotQA and substantially more on tasks requiring current information that was absent from the model's training data.

A correct answer reached in 3 steps is better than the same correct answer reached in 8 steps, because fewer steps means fewer tokens, lower latency, and lower cost. Efficiency is measured as the number of tool calls required to reach the correct answer, averaged across a test set. An agent that makes redundant or irrelevant searches has low trajectory efficiency even if it eventually reaches the right answer.

When the agent encounters an obstacle (a failed search, an ambiguous result, a calculation error), does it recover? You can measure this by deliberately introducing failures in the tool responses and observing whether the agent adapts. A well-designed agent should recover from a majority of single-step failures. Recovery from multi-step errors (where an early incorrect observation cascades into wrong subsequent actions) is harder and depends heavily on the model's ability to revisit assumptions.

The thought trajectory should accurately reflect the reasoning behind each action. An unfaithful trajectory might show a thought saying "I need to search for Hemingway's birthplace" while generating a completely unrelated action. Faithfulness can be measured by checking whether the action logically follows from the thought. While this is partially qualitative, it is important for interpretability and for building user trust.