Quality Monitoring › Quality Monitoring

Deploying a language model to production is not the end of the engineering work; it is closer to the beginning of a new phase. Once a model is serving live traffic, you need to know whether it is still doing its job. Outputs might degrade silently. The distribution of incoming requests can shift over weeks or months in ways that push the model outside the conditions it was optimized for. A prompt template that worked beautifully in evaluation may stop working once real users interact with it in ways the evaluation set never anticipated.

Quality monitoring is the discipline of continuously measuring what your model produces after deployment, detecting when quality degrades, and triggering the appropriate response before users notice a problem. It sits alongside infrastructure monitoring (covered in the previous chapter) but asks a fundamentally different question. Infrastructure monitoring asks: "Is the system running?" Quality monitoring asks: "Is the system producing good outputs?" A server can be fully healthy, with 99.9% uptime and sub-100ms latency, while the model inside it is generating responses that are wrong, incoherent, or harmful. Infrastructure metrics cannot see this. Only quality metrics can.

The challenge is that quality is expensive and ambiguous to measure. In evaluation, you typically hold out a labeled test set and score the model against it. In production, labels are rarely available in real time. You cannot pause traffic while a human annotator reads every response. You need automated signals that correlate with quality, statistical methods that distinguish real degradation from normal fluctuation, and alerting logic that escalates at the right moment without creating alert fatigue that desensitizes engineers.

This chapter covers the full quality monitoring lifecycle: what metrics to track, how to detect that those metrics are drifting or regressing, how to build alerting pipelines that fire at the right threshold, and how to architect a monitoring system that remains practical as your model and traffic scale. The techniques apply broadly across use cases, from customer service assistants to code generation tools to retrieval-augmented question answering, though the specific metric choices will differ by task.