Refining the Agent Using Observability › Connecting to Evaluation
Michael BrenndoerferAugust 10, 20251 min read
AI Agent HandbookAll chapters Part 12 of 15Observability and Debugging - Peering Inside the Agent
Refining the Agent Using Observability › Connecting to EvaluationOne section · Sections in this chapter6 / 11 ↓
Sections in this chapter11
Remember the evaluation framework from Chapter 11? Observability feeds directly into that process. Your logs provide the data you need to evaluate your agent's real-world performance.
For example, if you set a success criterion that "the agent should complete tasks in under 5 seconds," your logs tell you whether you're meeting that goal:
In[2]:
Codedef check_performance_goal(logs, max_response_time_ms=5000):
"""Check if agent meets response time goal."""
response_times = [log.get("response_time_ms", 0) for log in logs]
within_goal = sum(1 for t in response_times if t <= max_response_time_ms)
total = len(response_times)
success_rate = within_goal / total if total > 0 else 0
print(
f"Response time goal: {success_rate:.1%} of responses under {max_response_time_ms}ms"
)
print(f"Average response time: {sum(response_times) / total:.0f}ms")
return success_rate
success_rate = check_performance_goal(logs)
if success_rate < 0.90: # Goal: 90% of responses under 5 seconds
print("Need to optimize response time!")Your logs also help you create better test cases. When you see a query that the agent handled poorly, add it to your test suite. This ensures you don't regress when making future changes.