Refining the Agent Using Observability › Discovering Patterns in Agent Behavior
AI Agent HandbookAll chapters Refining the Agent Using Observability › Discovering Patterns in Agent BehaviorOne section · Sections in this chapter2 / 11 ↓
When you monitor your agent's logs over time, patterns emerge. These patterns tell you stories about how your agent actually works in practice, not just how you think it works.
Here are some common patterns you might discover:
Repeated "I don't know" responses: If your agent frequently says it doesn't know the answer to certain types of questions, that's a signal. Maybe it needs access to a new tool, or perhaps its knowledge base is missing key information.
Unnecessary tool calls: Sometimes an agent calls a tool when it doesn't need to. For example, it might search the web for information it already has in memory, or it might call a calculator for simple arithmetic it could handle directly.
Inefficient reasoning chains: You might notice the agent takes a long, winding path to reach conclusions that could be more direct. This wastes time and tokens.
Repeated wrong assumptions: If the agent consistently makes the same incorrect assumption in its chain-of-thought, that's a clear signal that something in its prompt or reasoning process needs adjustment.
Let's look at a practical example. Suppose you've been running your assistant for a week, and you review the logs:
## Example: Analyzing a week's worth of logs
import json
from collections import Counter
def analyze_logs(log_file):
"""Analyze patterns in agent logs."""
tool_calls = []
unknown_responses = []
reasoning_lengths = []
with open(log_file, "r") as f:
for line in f:
log = json.loads(line)
# Track tool usage
if "tool_call" in log:
tool_calls.append(log["tool_call"]["name"])
# Track "I don't know" responses
if "I don't know" in log.get("response", ""):
unknown_responses.append(log["query"])
# Track reasoning chain length
if "reasoning_steps" in log:
reasoning_lengths.append(len(log["reasoning_steps"]))
return {
"tool_usage": Counter(tool_calls),
"unknown_queries": unknown_responses,
"avg_reasoning_length": sum(reasoning_lengths) / len(reasoning_lengths)
if reasoning_lengths
else 0,
}
## Analyze the logs
patterns = analyze_logs("assistant_logs.jsonl")
print("Tool Usage:")
for tool, count in patterns["tool_usage"].most_common():
print(f" {tool}: {count} times")
print(
f"\nAverage reasoning chain length: {patterns['avg_reasoning_length']:.1f} steps"
)
print(
f"\nQueries resulting in 'I don't know': {len(patterns['unknown_queries'])}"
)
for query in patterns["unknown_queries"][:5]: # Show first 5
print(f" - {query}")This simple analysis might reveal that your assistant:
- Called the weather API 50 times but the calculator only 3 times
- Has an average reasoning chain of 4.2 steps
- Said "I don't know" to 12 queries, mostly about recent news events
Each of these observations suggests a potential improvement.