Model Routing › Model Routing

When you deploy a language model into production, you rarely have just one model. A modern production system might run a small 7B-parameter model alongside a 70B model and a frontier API like GPT-4 or Claude. The challenge is deciding which model should handle which request. Route everything to the big model and your costs explode. Route everything to the small model and quality suffers. Model routing is the discipline of making that decision intelligently: matching each request to the most appropriate model based on its complexity, cost constraints, latency requirements, and expected difficulty.

To appreciate why this matters, consider the economics of inference. Frontier models like GPT-4 or Claude cost roughly 10 to 50 times more per token than a well-tuned 7B open-source model. If you have 100,000 daily requests, even routing 70% of them to the cheaper tier while maintaining comparable quality on easy tasks can translate to 60-70% cost reductions. At scale, that difference often determines whether a product is economically viable.

This chapter covers model routing methods, including simple rule-based selection, learned routing classifiers, A/B tests for routing decisions, and cascaded systems that escalate requests through increasingly capable (and expensive) models. We also examine the tradeoffs between different routing strategies and the operational challenges of running routed systems at scale.