Testing the Agent with Examples › Testing Frameworks and Tools
AI Agent HandbookAll chapters Testing the Agent with Examples › Testing Frameworks and ToolsOne section · Sections in this chapter6 / 11 ↓
While we've built our own simple test runner, you don't have to start from scratch. Several tools can help you test AI agents more effectively.
Python Testing Frameworks:
The standard Python testing frameworks work well for agent testing. You can use pytest or unittest to structure your tests in a familiar way:
class TestAgentCapabilities:
def test_calculator_tool(self, agent):
response = agent.process("What's 1,234 multiplied by 5,678?")
assert "7,006,652" in response["text"]
assert "calculator" in response["actions_taken"]
def test_memory_storage(self, agent):
agent.process("Remember that my birthday is July 20th.")
response = agent.process("When is my birthday?")
assert "July 20" in response["text"].lower()Using pytest gives you powerful features like fixtures (for setting up test agents), parametrized tests (for running the same test with different inputs), and detailed failure reports. You can run your tests with pytest tests/ and get professional-grade test output.
LLM-Specific Testing Tools:
Several tools have emerged specifically for testing language model applications:
LangSmith (from LangChain) gives you tracing, evaluation, and testing in one package. You can create test datasets, run evaluations, and track your agent's performance across versions. If you're already using LangChain, this integrates naturally. Learn more at https://docs.smith.langchain.com/
LangFuse is an open-source observability and evaluation platform. You can trace execution, create test cases, and run automated evaluations. It works with various LLM frameworks and includes a dashboard for monitoring results. See https://langfuse.com/docs
Promptfoo specializes in testing LLM outputs. You define test cases in YAML files, run batch evaluations, and compare different prompts or models. This works well when you're iterating on prompt design. Visit https://promptfoo.dev/docs/intro
OpenAI Evals offers standardized benchmarks and custom evaluation tools for GPT-5 and other OpenAI models. You can build your own evals or use existing ones to measure specific capabilities. Check out https://github.com/openai/evals
Braintrust helps you evaluate and improve AI applications with tools for creating datasets, running experiments, and tracking metrics over time. See https://www.braintrust.dev/docs
These frameworks handle much of the infrastructure we built manually: running tests in parallel, tracking results over time, comparing versions, and generating reports. If you're building a production agent, they're worth exploring. However, understanding how testing works at the fundamental level (as we've done here) helps you use these tools more effectively.