Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Refining the Agent Using Observability › Observability in Production

Michael BrenndoerferAugust 10, 20252 min read

AI Agent HandbookAll chapters

Part 12 of 15Observability and Debugging - Peering Inside the Agent

Chapter 35Refining the Agent Using Observability › Observability in Production

Reading section 7 of 11 · Observability in Production · 2 minSection 7 of 11 · 2 min

Section 7 of 11 · 2 min read

Refining the Agent Using Observability › Observability in ProductionOne section · Sections in this chapter7 / 11 ↓

Sections in this chapter11

Once your agent is deployed and real users are interacting with it, observability becomes even more important. You're no longer just testing with your own queries. You're seeing how diverse users with different needs interact with your assistant.

In production, you want to:

Track aggregate metrics: Monitor overall performance, not just individual queries. Are response times trending up? Is the error rate increasing?

Set up alerts: If something goes seriously wrong (error rate spikes, response times exceed a threshold), you want to know immediately.

In[2]:

Code

def check_for_anomalies(logs, window_size=100):
    """Simple anomaly detection for production logs."""
    recent_logs = logs[-window_size:]

    # Check error rate
    errors = sum(1 for log in recent_logs if log.get("error"))
    error_rate = errors / len(recent_logs)

    if error_rate > 0.05:  # More than 5% errors
        print(f"⚠️  Alert: Error rate is {error_rate:.1%} (threshold: 5%)")

    # Check response times
    response_times = [log.get("response_time_ms", 0) for log in recent_logs]
    avg_time = sum(response_times) / len(response_times)

    if avg_time > 8000:  # Slower than 8 seconds
        print(
            f"⚠️  Alert: Average response time is {avg_time:.0f}ms (threshold: 8000ms)"
        )

Respect user privacy: When logging in production, be careful about what you record. Redact sensitive information like personal data, passwords, or confidential business information.

In[4]:

Code

import re
from datetime import datetime


def redact_sensitive_info(text):
    """Remove sensitive information from logs."""
    # Redact email addresses
    text = re.sub(
        r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "[EMAIL]", text
    )

    # Redact phone numbers
    text = re.sub(r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b", "[PHONE]", text)

    # Redact credit card numbers
    text = re.sub(r"\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b", "[CARD]", text)

    return text


## Use this when logging user queries
log_entry = {
    "query": redact_sensitive_info(user_query),
    "response": redact_sensitive_info(agent_response),
    "timestamp": datetime.now().isoformat(),
}