Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Monitoring and Reliability › Tracking Key Metrics

Michael BrenndoerferAugust 20, 20252 min read

AI Agent HandbookAll chapters

Part 14 of 15Deployment and Operations - From Prototype to Production

Chapter 40Monitoring and Reliability › Tracking Key Metrics

Reading section 5 of 14 · Tracking Key Metrics · 2 minSection 5 of 14 · 2 min

Section 5 of 14 · 2 min read

Monitoring and Reliability › Tracking Key MetricsOne section · Sections in this chapter5 / 14 ↓

Sections in this chapter14

Health checks tell you if the agent is alive, but they don't tell you how well it's performing. For that, you need metrics. Metrics are numerical measurements that track your agent's behavior over time.

Here are the most useful metrics to track:

Request Rate: How many requests is your agent handling per minute or per hour? This tells you how much the agent is being used.

Response Time: How long does it take the agent to respond to a request? Users notice if responses are slow, so tracking this helps you maintain good performance.

Error Rate: What percentage of requests result in errors? A sudden spike in errors indicates something is wrong.

Tool Usage: How often is each tool being called? This helps you understand what your agent is doing and can reveal if a tool is being overused or underused.

Let's add basic metrics tracking to our agent:

In[2]:

Code

import time
from collections import defaultdict

## Simple in-memory metrics storage
metrics = {
    "request_count": 0,
    "error_count": 0,
    "response_times": [],
    "tool_usage": defaultdict(int)
}

@app.route('/chat', methods=['POST'])
def chat():
    start_time = time.time()
    metrics["request_count"] += 1
    
    try:
        user_message = request.json.get('message')
        
        # Process the message with your agent
        response = process_agent_request(user_message)
        
        # Track response time
        response_time = time.time() - start_time
        metrics["response_times"].append(response_time)
        
        return jsonify({"response": response})
        
    except Exception as e:
        metrics["error_count"] += 1
        logger.error(f"Request failed: {e}")
        return jsonify({"error": "Something went wrong"}), 500

@app.route('/metrics', methods=['GET'])
def get_metrics():
    """Expose metrics for monitoring"""
    avg_response_time = (
        sum(metrics["response_times"]) / len(metrics["response_times"])
        if metrics["response_times"] else 0
    )
    
    return jsonify({
        "total_requests": metrics["request_count"],
        "total_errors": metrics["error_count"],
        "error_rate": metrics["error_count"] / max(metrics["request_count"], 1),
        "avg_response_time_seconds": avg_response_time,
        "tool_usage": dict(metrics["tool_usage"])
    })

Now you have a /metrics endpoint that shows how your agent is performing. You can check this periodically to see trends. Is the error rate increasing? Are response times getting slower? These signals help you catch problems before users complain.

For production systems, you'd typically use a proper metrics library like Prometheus or send metrics to a monitoring service like Datadog or New Relic. But the principle is the same: track the numbers that matter, and watch for changes.