Quality Monitoring › Quality Alerts

Detecting a problem is not the same as responding to it. A monitoring system that detects drift but takes no action, or sends an alert that nobody responds to, provides no actual protection. A well-designed alerting system notifies the right people at the right urgency level, provides enough context to diagnose the problem quickly, and integrates with workflows that can take remedial action.

Severity tiers distinguish between degradations that require immediate action and those that warrant investigation on a normal schedule. A common three-tier scheme uses different thresholds and different notification channels:

• Critical: Metrics fall outside bounds that indicate a broken deployment or severe user impact. Requires immediate on-call response, typically within minutes. Appropriate for hard failures like toxicity classifier exceeding a threshold or output length collapsing to near zero.
• Warning: Metrics show concerning trends that are not yet at critical levels. Requires investigation within hours. Appropriate for gradual drift that is accelerating or for PSI values in the "investigate" range.
• Informational: Metrics show notable changes that are within acceptable bounds but worth monitoring. Surfaced in dashboards and weekly reports rather than sent as pages. Appropriate for minor distribution shifts or slight metric variability increases.

Alert fatigue is the primary enemy of effective alerting. If alerts fire too often, on-call engineers begin ignoring them or muting notification channels, which defeats the purpose of monitoring entirely. The root cause of alert fatigue is almost always thresholds set too tightly relative to the natural variability of the metric, or metrics with high intrinsic variability being used as alerting signals without sufficient smoothing. Fighting alert fatigue requires monitoring alert rate explicitly and treating excessive alerts as a bug in the monitoring system itself. A useful heuristic: if an on-call engineer would not take a different action upon receiving an alert than they would without it, the alert is not pulling its weight.

Multi-metric alerts correlate several signals before firing. A drop in output quality scores is more alarming if it co-occurs with a shift in input embedding distribution than if it appears in isolation. The co-occurrence suggests a causal story: inputs changed, and quality followed. An isolated drop in output quality with no corresponding input shift might indicate model instability rather than distribution shift, which has a different remediation path. Building correlation logic into alert rules reduces false positives while preserving sensitivity to real problems. The trade-off is that correlation logic is more complex to implement and maintain.

Actionability is a requirement, not a nice-to-have. Every alert should have a clear owner and a documented response runbook that specifies what actions to take, in what order, and under what conditions to escalate. An alert that nobody knows how to respond to is worse than no alert, because it creates urgency without direction and depletes the on-call engineer's attention and trust in the monitoring system. Before deploying an alert, ask: what will the on-call engineer do when this fires? If the answer is "look at a dashboard and wonder what happened," the alert is not ready.

Setting thresholds well requires understanding the natural variability of your metrics under normal operating conditions. A common approach is to collect a baseline window of at least several weeks of production data, compute the mean and standard deviation of each metric, and set warning thresholds at mean minus two standard deviations and critical thresholds at mean minus three standard deviations. Under a Gaussian distribution, values more than two standard deviations below the mean occur about 2.3% of the time by chance; values more than three standard deviations below occur about 0.13% of the time. These rates provide a rough guide to expected false alarm frequency.

For metrics that have strong temporal patterns, such as quality scores that are lower during high-traffic periods because sampling effects mean fewer queries are labeled in real time, or scores that vary by day of week because user intent differs between weekdays and weekends, you should baseline separately by time-of-day and day-of-week rather than using a flat threshold. Failing to account for temporal patterns produces threshold crossings every night at peak traffic or every Monday when user query mix changes, generating exactly the kind of alert fatigue that undermines the monitoring system.

Percentile-based thresholds are less sensitive to outliers than mean-based thresholds for right-skewed or heavy-tailed distributions, which are common in quality metrics. Instead of alerting when the mean drops below a threshold, alerting when the 10th percentile drops below a threshold ensures you catch widespread degradation even when a small fraction of high-quality outputs is masking the problem in the mean. A 10th percentile threshold says: the bottom decile of your outputs should not be worse than this. This is often more directly interpretable in terms of user impact than a mean threshold.

Trend-based alerting detects degradation earlier by alerting on the slope of the metric rather than its absolute value. If quality has been declining at a rate of 0.005 BLEU per day for the past two weeks, that is operationally significant even if the current value has not yet crossed a fixed threshold. Trend-based alerts require fitting a simple regression to the recent metric history and testing whether the slope is significantly negative. The advantage is earlier detection; the disadvantage is that short-term fluctuations can produce spurious negative slopes, so trend alerts typically require the slope to be negative over a minimum duration.

Many of the most reliable quality signals come from user behavior rather than automated metrics. Users who abandon a session after receiving a response, click "thumbs down" on an output, submit a follow-up correction, or escalate to a human agent are providing implicit or explicit quality signals. Building infrastructure to collect, label, and route this feedback back into your quality monitoring system closes the loop between production performance and ground truth.

The challenge with user feedback is that it is sparse and biased. Most users do not provide explicit feedback, and those who do are not a representative sample: unhappy users are overrepresented (they are more likely to click "thumbs down"), and users with specific demographic or technical profiles are overrepresented relative to the general user population. Using user feedback as the sole quality signal will produce systematically biased quality estimates. Using it as one signal among many, calibrated against a representative human annotation sample, is more reliable.

Ground truth collection pipelines automate labeling for a sample of live outputs by routing them to human annotators or to a high-quality judge model. Even labeling 0.5% of outputs at a moderate daily traffic volume provides thousands of labeled examples per week, which is statistically powerful for quality estimation. The labeled sample serves multiple purposes: it provides ground truth for calibrating your automated metrics, checking for systematic biases between automated and human quality assessments, giving evidence for regression detection, and building an ongoing evaluation corpus that grows with your deployment.

The infrastructure for ground truth collection needs to address several practical concerns. Sampling strategy matters: uniform random sampling is simple but inefficient; stratified sampling that over-samples rare query types or unusual model outputs is more informative per label. Annotator consistency matters: inter-annotator agreement should be measured and annotators calibrated against each other to ensure quality labels are reliable. Pipeline latency matters: a ground truth pipeline that takes two weeks to return labels is too slow to detect fast-moving regressions. Some teams separate fast-path automated metrics from slow-path human evaluation, using the fast path for real-time alerting and the slow path for deep regression analysis.