Model Routing › Advanced Routing Architectures

The strategies described so far handle routing as a one-shot decision per request. Several more sophisticated architectures extend this foundation.

Speculative execution is a variant where the small model and large model run in parallel rather than sequentially. The large model begins generating a response at the same time as the small model. When the small model finishes, an evaluator assesses its output. If the small model's response is acceptable, the large model's generation is cancelled and the small model's response is returned. If not, the system waits for the large model to finish and returns its response.

This eliminates the latency penalty for escalated requests: instead of running small then large sequentially, the large model has been running the entire time. The cost penalty for speculation is the wasted large model compute on requests that ultimately did not need escalation. Speculation trades compute efficiency for latency guarantees.

The economic viability of speculative execution depends on the escalation rate and cost ratio. If 30% of requests escalate and the large model costs 10x the small model, then every 10 requests incurs the full large model cost for 10 requests but only 3 of them were necessary, wasting 7 large model runs. Whether this is worth the latency improvement depends on your latency requirements and cost budget.

Rather than a single routing decision, some systems maintain multiple specialized routers, each tuned for a different request type, and use a meta-router to select which router to apply. A meta-router might first classify the request domain (code, math, creative writing, factual lookup), then apply a domain-specific routing policy that is calibrated for that domain's difficulty distribution.

This hierarchical approach works because difficulty is not uniform across domains. A 7B model might handle creative writing requests as well as a 70B model in most cases, but struggle with mathematical reasoning even for moderately difficult problems. Routing policies calibrated on all request types together will systematically under-route in some domains and over-route in others. Domain-specific policies correct for this.

The downside is complexity: you now have $D \times T$ routing policies, where $D$ is the number of domains and $T$ is the number of model tiers. Maintaining, updating, and monitoring these policies scales linearly with the number of domains.

In production systems with hard cost budgets, routing policies must adapt dynamically to remaining budget. A system with a fixed monthly budget needs to be more conservative with the large model as the end of the month approaches.

Budget-constrained routing formalizes this by incorporating remaining budget directly into the routing decision. One approach modifies the cost-quality tradeoff parameter $\lambda$ dynamically:

$$\lambda(t) = \lambda_0 \cdot \frac{T}{B(t)}$$

where $T$ is the total monthly budget, $B(t)$ is the remaining budget at time $t$, and $\lambda_0$ is a base cost sensitivity. As budget is depleted, $\lambda(t)$ increases, pushing more traffic to the cheaper model. When budget is ample (early in the month), $\lambda(t)$ is low and the system prioritizes quality.

This creates a natural budget allocation strategy without requiring explicit planning. The system automatically becomes more conservative as constraints tighten, without any manual intervention.