Quality Monitoring › Summary

Quality monitoring gives you visibility into what your language model produces after deployment, not just what it produced during evaluation. The key ideas from this chapter are:

• Output quality degrades through several mechanisms: input distribution shift, label distribution shift, model regression from updates, feedback loop degradation, and knowledge staleness. Each mechanism has a different signature and requires different monitoring strategies.
• Reference-based metrics like BLEU, ROUGE, and BERTScore compare outputs to known-good references, giving direct quality estimates when ground truth is available. BLEU is cheap but sensitive to tokenization; BERTScore is more expensive but captures semantic equivalence.
• Reference-free metrics like self-consistency, confidence calibration, LLM-as-judge scoring, and task-completion proxies estimate quality without ground truth. They cover a larger fraction of production traffic but are noisier and require their own calibration.
• Semantic drift metrics including output embedding distribution, vocabulary statistics, and sentiment scores measure how outputs are changing over time, often providing early warning before quality-labeled metrics detect a problem.
• Drift detection applies statistical tests to determine whether current quality metrics have shifted from their baseline. The KS test operates on fixed windows and is sensitive to distributional shape; CUSUM and ADWIN operate sequentially and offer lower detection latency for location shifts.
• PSI quantifies distribution shift between a reference period and the current period, giving a scalar summary with interpretable thresholds: below 0.1 is stable, 0.1 to 0.25 warrants investigation, above 0.25 requires action.
• Regression detection uses A/B testing with appropriate statistical tests to determine whether a deliberate model change has caused a statistically significant quality degradation. Sliced evaluation reveals regressions in subgroups that aggregate metrics would hide.
• A well-designed alert system uses severity tiers, multi-metric correlation, carefully calibrated thresholds, and documented runbooks to balance sensitivity against alert fatigue and ensure every alert drives a meaningful response.
• Monitoring is most effective when paired with human calibration of automated metrics, fast-path rule-based guards for critical quality dimensions, baseline management that accounts for intentional model updates, and tight integration with deployment rollback workflows.