Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Testing the Agent with Examples › Building Your Test Suite

Michael BrenndoerferAugust 2, 20252 min read

AI Agent HandbookAll chapters

Part 11 of 15Evaluation - Measuring Your Agent's Performance

Chapter 31Testing the Agent with Examples › Building Your Test Suite

Reading section 2 of 11 · Building Your Test Suite · 2 minSection 2 of 11 · 2 min

Section 2 of 11 · 2 min read

Testing the Agent with Examples › Building Your Test SuiteOne section · Sections in this chapter2 / 11 ↓

Sections in this chapter11

A good test suite covers the full range of your agent's capabilities. For our personal assistant, that means creating examples that exercise each of the skills we've built: answering questions, using tools, remembering information, and planning multi-step tasks.

Let's start simple. Here's what a basic test case looks like:

In[3]:

Code

test_case = {
    "id": "math_001",
    "input": "What's 1,234 multiplied by 5,678?",
    "expected_behavior": "uses_calculator",
    "expected_output": "7,006,652",
    "category": "tool_use",
}

Each test case captures three essential pieces:

What you're asking the agent to do (the input)
How you expect it to approach the task (the behavior)
What result you want to see (the output)

Notice that we're not just checking if the agent gets the right answer. We're also verifying that it uses the right approach. For a math problem, we expect the agent to use its calculator tool rather than guessing. This distinction matters because it tells us whether the agent is reasoning correctly, not just getting lucky.

Let's build a small test suite that covers different aspects of our assistant's capabilities:

In[4]:

Code

## Using Claude Sonnet 4.5 for its superior agent reasoning and tool use
test_suite = [
    {
        "id": "math_001",
        "input": "What's 1,234 multiplied by 5,678?",
        "expected_behavior": "uses_calculator",
        "expected_output": "7,006,652",
        "category": "tool_use",
    },
    {
        "id": "memory_001",
        "input": "Remember that my birthday is July 20th.",
        "expected_behavior": "stores_to_memory",
        "expected_output": "confirmation of storage",
        "category": "memory",
    },
    {
        "id": "memory_002",
        "input": "When is my birthday?",
        "expected_behavior": "retrieves_from_memory",
        "expected_output": "July 20th",
        "category": "memory",
        "requires": ["memory_001"],  # This test depends on a previous one
    },
    {
        "id": "reasoning_001",
        "input": "If I have a meeting at 2 PM and it takes 45 minutes to get there, when should I leave?",
        "expected_behavior": "step_by_step_reasoning",
        "expected_output": "1:15 PM",
        "category": "reasoning",
    },
    {
        "id": "planning_001",
        "input": "Find today's weather and recommend what I should wear.",
        "expected_behavior": "uses_weather_tool_then_makes_recommendation",
        "expected_output": "weather report followed by clothing suggestion",
        "category": "planning",
    },
]

This suite tests five different capabilities. We have a math calculation that requires tool use, memory storage and retrieval, a reasoning problem, and a planning task that involves multiple steps. Notice how some tests depend on others. The second memory test only works if the first one successfully stored the birthday information.