Monitoring › Alerting Philosophy and Practices

The most important property of a good alert is that it requires action. An alert that operators routinely acknowledge and dismiss without doing anything is a broken alert. It trains operators to ignore the system and erodes trust in the entire monitoring stack. Before adding an alert, ask: if this fires at 3 AM, what action would the on-call engineer take? If the honest answer is "look at it, see nothing actionable, and go back to sleep," the alert should either be converted to a non-paging notification or removed entirely.

This principle has a name in the SRE literature: alerts should be symptoms, not causes. An alert on high CPU usage is a cause-based alert: it tells you something about the internals of the system but not whether users are affected. An alert on high p99 latency is a symptom-based alert: it directly measures what users experience. Prefer symptom-based alerts for your paging tier and use cause-based alerts only for warning-level notifications that inform investigation rather than demand immediate response.

For LLM services, a practical tier of alerts looks like the following. Critical alerts, which wake someone up, cover complete service outages (model backend unreachable), catastrophic error rates above 10%, and safety classifier failures where the monitoring infrastructure itself is down. Warning alerts, which notify but do not wake, cover error rates between 2-10%, p99 latency above service-level objectives, and quality score degradation below the defined threshold. Informational alerts cover cost anomalies (token consumption significantly above baseline) and slow quality drift that does not yet require action but should be tracked.

Alert thresholds deserve the same rigor as any other engineering decision. A threshold that is too tight generates noise; one that is too loose misses real problems. The right starting point is usually to set thresholds based on historical distributions: alert when a metric falls more than two or three standard deviations from its 30-day rolling mean. This adapts to seasonal patterns and gradual growth without requiring manual tuning. Absolute thresholds (error rate above 5%) are simpler to understand but require re-tuning as the system evolves.

Runbooks are the underrated half of good alerting. An alert fires. An on-call engineer wakes up. What do they do? A well-written runbook describes the alert in plain English, explains what the metric means, lists the most common causes (in order of frequency), describes the diagnostic steps to take first, and provides links to relevant dashboards and commands. A runbook written when you set up the alert is far more useful than one written from memory at 2 AM during an incident. Teams that invest in runbooks find that their mean time to resolution drops significantly, because junior engineers can handle incidents that would previously have required escalation to senior staff.