Model Routing › A/B Testing for Routing Decisions

A/B testing is the standard method for evaluating whether a routing strategy improves outcomes. Without rigorous A/B testing, it is easy to believe a routing improvement is real when it is just noise, or to miss an improvement because the measurement methodology was flawed.

The stakes are high. Routing changes can silently degrade quality for a subset of users without triggering any obvious error signals. Unlike a bug that causes crashes, a routing policy that routes 5% of hard requests to the weak model manifests as a small quality drop that is easy to rationalize as noise. Rigorous A/B testing is the discipline that separates real improvements from wishful thinking.

An A/B test for model routing compares two routing strategies:

• Control (A): The current routing policy, typically a single model or an existing routing rule.
• Treatment (B): The new routing policy being evaluated.

Incoming requests are randomly split between the two strategies. The split must be random at the request level to ensure statistical independence. Splitting by user (all requests from a given user go to the same strategy) helps detect personalization effects but requires careful analysis to account for user-level confounders. User-level splitting is appropriate when routing decisions interact with user history or when the quality of a response depends on prior responses in the same session.

Key metrics to measure:

• Quality: Measured by human evaluation, automated scoring, or task-specific metrics (BLEU, accuracy, etc.)
• Latency: P50, P90, and P99 latencies for each strategy
• Cost per request: Total inference cost divided by request count
• Escalation rate (for cascades): Fraction of requests reaching each tier

The choice of quality metric deserves careful attention. Automated metrics are cheap and scalable but may not align with what users care about. Human evaluation is expensive but provides ground truth. A common hybrid is to use automated metrics for continuous monitoring and human evaluation for final decision-making. The automated metric should be validated against human judgments before you trust it to make routing decisions.

Latency metrics require percentile reporting, not just mean latency. Mean latency is heavily influenced by slow outliers and can mask important patterns. A routing strategy that reduces mean latency by 10% while increasing P99 latency by 50% is not an improvement from a user experience perspective. P99 latency determines the worst experience users encounter in normal operation.

A/B test results should not be trusted until they achieve statistical significance. For quality metrics scored as continuous values, a two-sample $t$-test compares means. For binary quality metrics (pass/fail), a proportion test or chi-squared test is appropriate.

The minimum detectable effect (MDE) depends on sample size, variance, and the significance threshold $\alpha$. For a routing test where quality improvements may be small (1-2%), large sample sizes are needed. A useful rule of thumb: if your daily traffic is $N$ requests and you split 50/50, you need approximately:

$$n \approx \frac{2 \cdot z_{\alpha/2}^2 \cdot \sigma^2}{\delta^2}$$

where:

• $z_{\alpha/2}$: the critical value for significance level $\alpha$ (for $\alpha = 0.05$, $z_{\alpha/2} \approx 1.96$)
• $\sigma^2$: the variance of the quality metric
• $\delta$: the minimum effect size you want to detect

At $\alpha = 0.05$ and $\delta = 0.02$ (2% quality improvement) with $\sigma = 0.3$, you need roughly 1,700 samples per arm. A system with 10,000 daily requests could complete this test in under a day. A system with 100 daily requests would need three weeks.

This formula highlights a real tension for early-stage products. You cannot get fast feedback on routing quality until you have substantial traffic. This is one reason routing systems are typically deployed later in a product's lifecycle, after traffic has grown enough to support rapid experimentation.

Multiple testing is another concern. If you run many routing experiments simultaneously, some will appear significant by chance. The Bonferroni correction reduces each experiment's significance threshold by the number of simultaneous tests. A system running five experiments simultaneously at $\alpha = 0.05$ should apply a per-experiment threshold of $0.05 / 5 = 0.01$ to maintain the family-wise error rate. In practice, most routing systems run sequential rather than simultaneous experiments, which reduces but does not eliminate this concern.

A/B tests for routing policies need guardrails to prevent harm during the test. If the treatment strategy is dramatically worse, you want to detect this quickly and stop the experiment. Common guardrails include:

• Early stopping rules: Stop the experiment if quality metrics fall below a threshold in the treatment arm. Sequential testing methods like the Bonferroni-corrected sequential probability ratio test (SPRT) allow you to stop early with controlled error rates.
• Sample size caps: Limit the fraction of traffic exposed to the treatment arm (e.g., 5% treatment, 95% control) until initial results are promising. A bad treatment only harms 5% of users until you have confidence it is safe.
• Real-time monitoring: Watch latency and error rate dashboards during the test to catch operational issues immediately. A treatment strategy that introduces unexpected model calls or failures should be caught by alerting before it reaches statistical significance.
• Novelty filters: Ensure the test traffic is representative. If your treatment arm accidentally receives more hard requests due to non-random assignment, the apparent quality drop is confounded with traffic composition, not causally related to the routing strategy.

Some routing improvements take time to evaluate properly. If a new routing strategy routes certain users more consistently to the large model, those users may notice a quality improvement and engage more deeply with the product, which affects long-term metrics like retention and satisfaction. These effects are invisible in a short A/B test.

A holdout group is a permanently held-out fraction of traffic (typically 5-10%) that receives the baseline strategy even after the new strategy has been rolled out to everyone else. This allows long-term comparison of users who experienced the new strategy versus those who did not. Holdout groups are standard practice at large tech companies for exactly this reason: they catch effects that short experiments miss.

For routing systems specifically, holdout groups help measure whether routing improvements that look good on immediate quality metrics also improve downstream outcomes like task completion rate, follow-up request rate, or user satisfaction scores.