Quality Monitoring › Regression Detection

Drift detection tells you that something has changed. Regression detection tells you whether a deliberate change, such as a new model version or updated prompt template, has hurt quality. These are related but distinct problems. Drift detection operates on the ongoing stream of production data; regression detection is triggered at the moment of a model update and asks whether the new model is better or worse than the old one on a defined set of quality criteria.

The simplest form of regression detection is A/B testing: route a fraction of traffic to the new model and a fraction to the current model, collect quality metrics on both, and apply a statistical test to determine whether the difference is significant. The appropriate test depends on the metric type. For continuous metrics like BLEU score or embedding similarity, a t-test (for normally distributed metrics) or Mann-Whitney U test (for non-normal distributions) is appropriate. For binary outcomes like task-completion rate, a z-test for proportions or a chi-square test works. For bounded metrics with non-normal distributions, non-parametric alternatives are generally safer because quality metrics often have skewed or bounded distributions that violate the normality assumption.

The required sample size to detect an effect of size $d$ with power $1 - \beta$ at significance level $\alpha$ is:

$$n = \frac{2\sigma^2 (z_{\alpha/2} + z_\beta)^2}{d^2}$$

where:

• $\sigma^2$: variance of the metric under the null hypothesis, estimated from historical data
• $z_{\alpha/2}$: critical value for the significance level (e.g., 1.96 for $\alpha = 0.05$)
• $z_\beta$: critical value for the desired power (e.g., 0.84 for 80% power)
• $d$: minimum detectable effect, the smallest quality difference you consider operationally significant

This formula makes explicit the fundamental tension in regression testing: detecting smaller regressions requires more data, which means running the experiment longer, which means more users are exposed to a potentially worse model during the test period. A regression of 0.01 BLEU points may require tens of thousands of queries to detect reliably; a regression of 0.05 BLEU points may be detectable in hundreds. Choosing $d$ is a product decision: how small a regression is too small to care about? Setting $d$ too small makes experiments impractical; setting it too large risks missing regressions that users will notice.

One important consideration is whether to use a one-sided or two-sided test. For regression detection specifically, you are asking whether the new model is worse, not whether it is different. A one-sided test has more power to detect a regression of a given size than a two-sided test, because it concentrates the entire rejection region on one tail. The cost is that it cannot detect improvements, but regression detection is not trying to detect improvements; it is trying to prevent regressions from being deployed.

Aggregate metrics can hide regressions in important subgroups. A new model might improve performance on the most common query types while badly regressing on rare but critical ones. This is one of the most dangerous patterns in model updates: the aggregate metric looks better, so the update ships, but a specific user population experiences substantially worse quality. Sliced evaluation partitions traffic by query type, user segment, language, complexity level, or other dimensions and tests each slice independently.

Designing good slices requires domain knowledge. For a multilingual system, language is an obvious and critical slice. For a code generation system, programming language and task complexity (completing a function signature versus implementing an algorithm from scratch) are natural slices. For a dialog system, first-turn queries versus follow-up queries often behave differently because follow-up queries rely on conversation history in ways that the model may handle inconsistently. For a question-answering system, factual questions versus opinion questions versus multi-hop reasoning questions may respond differently to model changes.

You should define your slices before deployment rather than searching for regressions after the fact. Post-hoc slice discovery inflates false positive rates through multiple comparisons: if you test 50 slices and any one of them regresses at p < 0.05, you expect 2.5 spurious regression detections even under the null hypothesis. Defining slices in advance, or applying a multiple comparisons correction like Bonferroni correction when testing many slices, keeps the false positive rate under control.

Minimum slice performance constrains that no slice should regress beyond a specified threshold, even if aggregate performance improves. This is a more conservative requirement than aggregate improvement and is appropriate when certain subgroups are business-critical, legally protected, or represent user populations that are harder to reach if they churn. A model that improves aggregate BLEU by 0.02 while regressing on non-English queries by 0.05 should not ship, even though its aggregate score improved, if the non-English user population is important.

Before routing any live traffic to a new model, shadow mode runs the new model in parallel on all incoming requests but serves only the old model's outputs to users. The new model's outputs are logged and evaluated offline. This lets you measure regression risk on realistic production traffic with zero user impact during the evaluation period.

Shadow mode provides several advantages over A/B testing. It eliminates user exposure risk during evaluation. It allows evaluation on the full production distribution rather than the fraction of traffic routed to the treatment arm. It can be run for as long as needed to accumulate sufficient statistical power without extending user exposure to a potentially worse model. The main cost is that it requires running both models simultaneously, which roughly doubles inference cost. For models that are expensive to run, this may be prohibitive. For models where inference cost is modest relative to the risk of a bad deployment, shadow mode is the gold standard for regression evaluation.

A practical consideration is that shadow mode cannot measure quality dimensions that require actual user interaction to observe. If quality partly depends on how users respond to and follow up on model outputs, shadow mode will miss those dimensions. For these cases, a small A/B experiment with careful monitoring may be necessary even when shadow mode is the primary evaluation method.