Model Routing › Model Cascades

A model cascade is a sequential routing architecture where requests are processed by a cheap model first, and only escalated to more expensive models if the cheap model's output is deemed insufficient. This is sometimes called "progressive evaluation" or "cost-aware inference."

The fundamental difference from upfront routing classifiers is the direction of information flow. A routing classifier decides before generating any response. A cascade generates a response first and then decides whether that response is good enough. This allows the cascade to use the actual model output, including its confidence and uncertainty, as evidence for the escalation decision. You are not predicting whether the model will fail; you are observing whether it failed and acting accordingly.

A two-tier cascade works as follows. When a request arrives, the small model generates a response. An evaluator then assesses the small model's confidence or output quality. If the evaluator judges the output acceptable, it is returned to the user. If not, the request is forwarded to the large model, which generates its own response. The large model's output is returned as the final answer.

The evaluator is the critical component. Common evaluation strategies include:

• Self-confidence: The small model outputs a probability or confidence score with its response. Responses below a threshold are escalated.
• Proxy judge: A separate lightweight model evaluates the small model's output quality.
• Semantic uncertainty: Methods that estimate how uncertain the model is, such as measuring the entropy of the output distribution or generating multiple samples and measuring their variance.
• Factual consistency checks: For structured outputs, automated checks can verify whether the response satisfies basic constraints (valid JSON, correct format, required fields present). A response that fails these checks is escalated without further evaluation.

The quality of the cascade depends entirely on the quality of the evaluator. A poor evaluator that keeps low-quality outputs from the small model will degrade end-to-end quality, while an overly conservative evaluator that escalates everything will eliminate the cost savings. The evaluator is where most of the engineering effort in a cascade system goes.

Notice that a cascade imposes an inherent latency penalty on escalated requests. When the large model eventually handles a request, the system has already paid for the small model's generation plus the evaluator's assessment. The total latency for an escalated request is the sum of small model latency, evaluator latency, and large model latency. This means cascades are most economically efficient when a large fraction of requests are handled at the first tier and only a small minority escalate.

The most common cascade evaluator uses the model's own confidence as a proxy for output quality. For classification tasks, this is straightforward: if the model's maximum class probability exceeds a threshold $\tau$, return the response. Otherwise, escalate.

For generative tasks, confidence estimation is harder. Language models do not naturally output a single confidence score for a free-form generated response. Common proxies include:

• Token-level log probability: The sum or mean of log probabilities assigned to the generated tokens. Low log probability suggests the model is uncertain about its own output. The mean log probability per token, sometimes called the average negative log-likelihood, is particularly useful because it normalizes for response length, allowing fair comparison across responses of different lengths.
• Entropy of the first token distribution: High entropy on the first generated token often correlates with overall response uncertainty. This signal is especially cheap to compute because you only need a single forward pass to the first generation step, not the full response.
• Self-consistency: Generate $k$ responses from the small model and measure agreement. If $k$ diverse samples agree on the key facts or conclusion, the model is likely confident. High variance across samples signals uncertainty.

The self-consistency approach is particularly powerful for reasoning tasks. For a question with a single correct answer, generating five samples and checking whether they all agree provides a stronger confidence signal than any single probability score. A model that answers "42" five times out of five is almost certainly correct. A model that produces five different answers signals uncertainty. The downside is cost: generating $k$ samples multiplies the small model cost by $k$, which only makes sense if $k \cdot c_{\text{small}} \ll c_{\text{large}}$.

For each of these proxies, you select a threshold $\tau$ by calibrating on a validation set with known quality labels. The threshold trades off between:

• Escalation rate: The fraction of requests forwarded to the large model (directly determines cost).
• Quality coverage: The fraction of requests where the small model's output meets quality requirements.

This tradeoff has a characteristic shape. At very low thresholds, almost nothing escalates: you accept every small model response, quality is whatever the small model can produce, and cost is minimized. As you raise the threshold, more requests escalate, quality improves because you are catching the small model's worst outputs, but cost increases. At very high thresholds, almost everything escalates regardless of actual quality, and your cascade degenerates into always-large routing. The optimal threshold sits somewhere in the middle, depending on your quality requirements and budget.

Selecting the threshold requires a validation set with ground-truth quality labels. You sweep the threshold from 0 to 1, compute the quality-cost curve, and select the threshold that satisfies your quality constraint at minimum cost, or that maximizes quality given a cost budget. This calibration step is not optional. An uncalibrated threshold is as likely to hurt you as help you.

The cascade concept extends naturally to more than two tiers. A three-tier system might use a 7B model, then a 70B model, then a frontier API. Each tier applies its confidence check; only requests that exceed the escalation threshold at tier $i$ are forwarded to tier $i+1$.

The cost model for a $K$-tier cascade is:

$$\text{Expected Cost} = \sum_{k=1}^{K} c_k \cdot p_k$$

where:

• $c_k$: the cost per request for model tier $k$
• $p_k$: the probability that a request reaches tier $k$

For a well-calibrated cascade, $p_1 = 1$ (all requests hit tier 1), and $p_k < p_{k-1}$ for each subsequent tier (fewer and fewer requests escalate). If 80% of requests are resolved at tier 1, 15% at tier 2, and 5% at tier 3, you pay $c_1$ for 100% of requests, $c_2$ for 20%, and $c_3$ for 5%.

With realistic cost ratios (e.g., $c_1 = 1$, $c_2 = 10$, $c_3 = 100$), the expected cost is $1 \cdot 1.0 + 10 \cdot 0.2 + 100 \cdot 0.05 = 8.0$ units, compared to $100$ units if everything went to tier 3. That is a 12.5x cost reduction for the same end-to-end quality on the 80% of requests handled at tier 1.

Multi-tier cascades become progressively harder to calibrate. Each tier needs its own confidence estimator tuned to detect the specific failure modes of that tier's model. The thresholds interact: setting tier 1's threshold too low offloads requests to tier 2 that would have been fine, and they may not escalate further to tier 3 even if they should. This interdependency means you generally need to calibrate thresholds jointly rather than independently.

A practical simplification is to treat each tier's evaluator as independent and tune thresholds greedily: first optimize tier 1's threshold to minimize the number of easy requests that escalate while catching the cases where tier 1 fails, then optimize tier 2's threshold similarly for requests that reach it. This greedy approach does not find the global optimum but is much more tractable and works well in practice.

Cascades are not universally superior to upfront routing. There are several scenarios where they underperform.

When the small model is very slow, the latency penalty from always running the small model first may be unacceptable. If your small model takes 300 milliseconds and your large model takes 800 milliseconds, an escalated request takes over a second just for the small model's wasted inference. Users will notice.

When the escalation evaluator is weakly calibrated, the cascade may produce worse quality than either a good upfront classifier or always using the large model. A bad evaluator that fails to detect low-quality small model outputs produces an end-to-end system that silently degrades.

When almost all requests need the large model (high inherent difficulty distribution), a cascade wastes the cost of running every request through the small model first. If 80% of requests will escalate, the small model run is pure overhead.

Understanding these failure modes helps you decide when to use a cascade versus a routing classifier. A routing classifier is better when you have good upfront signals about difficulty. A cascade is better when difficulty is hard to predict before generation but easy to assess after.