Multi-Agent Systems › Multi-Agent Benchmarks

Measuring multi-agent system performance requires task benchmarks that expose coordination quality alongside final output quality. A benchmark that only measures whether the final answer is correct misses important information about how efficiently agents coordinate, how gracefully they handle failures, and whether the multi-agent setup contributes value beyond what a single agent would achieve.

Several benchmarks are designed specifically for multi-agent evaluation:

HotpotQA multi-hop reasoning (Yang et al., 2018): Questions that require gathering information from multiple sources before answering. A multi-agent system should outperform a single agent because specialized retrieval agents can focus on different evidence sources simultaneously. The benchmark's characteristic feature is that no single document contains the full answer, forcing information synthesis across sources.

WebArena (Zhou et al., 2023): A web browser benchmark testing agents' ability to complete realistic web tasks. Multi-agent versions test whether an orchestrator and web-browsing subagents can coordinate on complex sequences of web interactions. The benchmark is notable for testing long-horizon task completion where agents must maintain goals across many intermediate steps.

SWE-bench (Jimenez et al., 2023): Software engineering tasks requiring code editing to fix bugs in real repositories. Multi-agent approaches (separate agents for test reproduction, code editing, and verification) have shown improvements over single-agent baselines. The benchmark is challenging because it requires understanding large codebases and making targeted edits without breaking existing functionality.

CAMEL (Li et al., 2023): A role-playing framework specifically for evaluating agent communication and collaboration, where two agents (instructor and assistant) work together to complete tasks through conversation. CAMEL benchmarks measure task success and the quality of the inter-agent dialogue, including how well agents stay on task, handle misunderstandings, and reach consensus.

Multi-agent system evaluation should measure more than task success rate:

• Coordination overhead: How many inter-agent messages are required per task? Systems with excessive back-and-forth may be over-communicating.
• Parallelism utilization: What fraction of agent-seconds are spent waiting on dependencies? High wait times suggest the decomposition is not exploiting available parallelism.
• Recovery rate: How often do failed subtasks recover successfully? Low recovery rates indicate fragile agents or insufficient retry logic.
• Redundancy effectiveness: When multiple agents tackle the same problem for verification, how often does the verifier catch real errors vs. false positives?
• Latency vs. accuracy tradeoff: Does adding more agents improve accuracy, and at what latency cost? The answer should guide deployment decisions.

Tracking these metrics together reveals whether a multi-agent system is well-designed. A system with high accuracy but very high coordination overhead might be improved by reducing unnecessary communication. A system with low parallelism utilization likely has a suboptimal dependency structure where tasks that could run in parallel are constrained to run sequentially.

A common pattern for fair comparison: run the same task with a single powerful agent using the same total compute budget (time or token budget) as the multi-agent system, and compare final outputs. This controls for the "more resources" confound. If a multi-agent system uses five agent-calls where a single agent uses one, compare to a single agent with five calls' worth of thinking time (perhaps through chain-of-thought or self-critique loops).

Recent work has found that multi-agent debate (where agents criticize each other's answers) outperforms both single-agent responses and single-agent self-critique on mathematical and reasoning benchmarks. Disagreement forces revision rather than mere repetition. Critically, the improvement is most pronounced on problems where there is a clear right answer that can be verified: when there is no ground truth to converge toward, multi-agent debate can amplify shared errors rather than correcting them.