Refining the Agent Using Observability › Quantitative Metrics from Logs
AI Agent HandbookAll chapters Refining the Agent Using Observability › Quantitative Metrics from LogsOne section · Sections in this chapter4 / 11 ↓
Logs aren't just qualitative stories. They can provide quantitative metrics that help you track your agent's performance over time. These metrics connect directly to the evaluation concepts we covered in Chapter 11.
Here are some useful metrics you can extract from logs:
Tool retry rate: How often does the agent need to retry a tool call because it failed the first time? A high retry rate might indicate unreliable tools or poor error handling.
def calculate_retry_rate(logs):
"""Calculate how often tool calls need to be retried."""
total_tool_calls = 0
retried_calls = 0
for log in logs:
if "tool_call" in log:
total_tool_calls += 1
if log.get("retry_count", 0) > 0:
retried_calls += 1
return retried_calls / total_tool_calls if total_tool_calls > 0 else 0
retry_rate = calculate_retry_rate(logs)
print(f"Tool retry rate: {retry_rate:.1%}")Average reasoning length: How many steps does the agent typically take to reach a conclusion? This can indicate efficiency.
def calculate_avg_reasoning_length(logs):
"""Calculate average number of reasoning steps."""
lengths = [
len(log.get("reasoning_steps", []))
for log in logs
if "reasoning_steps" in log
]
return sum(lengths) / len(lengths) if lengths else 0
avg_length = calculate_avg_reasoning_length(logs)
print(f"Average reasoning chain length: {avg_length:.1f} steps")Response time by query type: How long does the agent take to respond to different types of queries? This helps identify bottlenecks.
def analyze_response_times(logs):
"""Break down response times by query type."""
from collections import defaultdict
times_by_type = defaultdict(list)
for log in logs:
query_type = log.get("query_type", "unknown")
response_time = log.get("response_time_ms", 0)
times_by_type[query_type].append(response_time)
# Calculate averages
avg_times = {
qtype: sum(times) / len(times) for qtype, times in times_by_type.items()
}
return avg_times
response_times = analyze_response_times(logs)
for query_type, avg_time in sorted(
response_times.items(), key=lambda x: x[1], reverse=True
):
print(f"{query_type}: {avg_time:.0f}ms")These metrics give you concrete numbers to track. You can set goals like "reduce average reasoning length from 4.2 to 3.5 steps" or "decrease tool retry rate from 8% to under 5%."