Continuous Feedback and Improvement › Tracking Progress Over Time
AI Agent HandbookAll chapters Continuous Feedback and Improvement › Tracking Progress Over TimeOne section · Sections in this chapter6 / 12 ↓
It helps to track metrics over time so you can see if your agent is actually getting better. Here are some metrics worth monitoring:
Success rate: What percentage of interactions end with positive user feedback or successful task completion? Track this weekly or monthly. You want this number going up.
Error frequency: How often do specific types of errors occur? For example, "tool call failures," "misunderstood queries," or "incomplete responses." You want these going down.
Response quality scores: If you're using AI evaluation, track the average quality score over time.
User satisfaction: If you collect user ratings, track average satisfaction over time.
Here's a simple script to track these metrics:
import json
from collections import defaultdict
from datetime import datetime, timedelta
def analyze_weekly_metrics(log_file, weeks=4):
"""Analyze metrics over the past N weeks."""
# Group interactions by week
weekly_data = defaultdict(lambda: {"total": 0, "successful": 0, "errors": defaultdict(int)})
with open(log_file, "r") as f:
for line in f:
entry = json.loads(line)
timestamp = datetime.fromisoformat(entry["timestamp"])
week = timestamp.strftime("%Y-W%W")
weekly_data[week]["total"] += 1
if entry["success"]:
weekly_data[week]["successful"] += 1
if "error_type" in entry:
weekly_data[week]["errors"][entry["error_type"]] += 1
# Print summary
print("Weekly Performance Metrics\n")
for week in sorted(weekly_data.keys())[-weeks:]:
data = weekly_data[week]
success_rate = (data["successful"] / data["total"] * 100) if data["total"] > 0 else 0
print(f"{week}:")
print(f" Total interactions: {data['total']}")
print(f" Success rate: {success_rate:.1f}%")
if data["errors"]:
print(f" Most common error: {max(data['errors'].items(), key=lambda x: x[1])}")
print()
## Run the analysis
analyze_weekly_metrics("agent_feedback.jsonl")Weekly Performance Metrics 2025-W48: Total interactions: 1 Success rate: 0.0%
This might output something like:
Weekly Performance Metrics
2025-W44:
Total interactions: 127
Success rate: 78.0%
Most common error: ('relative_time_error', 15)
2025-W45:
Total interactions: 143
Success rate: 84.6%
Most common error: ('tool_call_failure', 8)
2025-W46:
Total interactions: 156
Success rate: 89.1%
Most common error: ('tool_call_failure', 7)
2025-W47:
Total interactions: 168
Success rate: 91.7%
Most common error: ('response_too_long', 5)You can see the agent improving. Success rate went from 78% to 91.7% over four weeks. The relative time errors (which were most common in week 44) decreased after improvements were made. Now a different issue (responses being too long) is emerging as something to address next.