Model Routing › Why Routing Matters

A single model cannot simultaneously minimize latency, maximize quality, and minimize cost. These goals are in direct tension:

• Larger models produce higher quality outputs but require more memory, take longer to respond, and cost more per token to run.
• Smaller models respond quickly and cheaply but may struggle with complex reasoning, instructions with subtle constraints, or specialized domains.
• Specialized models excel in their domain but fail outside it.

Routing resolves this tension by treating the model selection itself as a decision problem. Instead of picking one model for all traffic, you build a system that dynamically assigns each request to the model best suited for it.

The economic advantage follows from the traffic mix. Suppose 80% of your traffic consists of simple factual lookups or short responses that a 7B model handles correctly, while only 20% requires deep reasoning that demands a 70B model. A naive system that routes everything to the 70B model pays full cost for all requests. A well-calibrated router that correctly classifies that 80/20 split reduces inference costs by roughly 4x while maintaining overall quality. For a startup paying $50,000 per month on inference, that improvement drops the bill to $12,500 without changing the user experience.

Quality imposes the opposite constraint. You could cut costs by routing everything to the cheapest model, but that sacrifices quality on the hard requests your users care most about. Routing lets you invest inference budget where it matters: applying heavy computational resources precisely where simpler models would fail.

A third motivation is latency. Users notice when responses take longer than two seconds. A small model that responds in 100 milliseconds delivers a fundamentally different user experience than a large model that takes three seconds. Routing short, straightforward queries to the fast small model while reserving the slower large model for complex tasks lets you hit latency targets that no single model can consistently achieve.

Before building a router, you need a working theory of what makes some requests harder than others. No single observable feature answers this question reliably.

Request difficulty correlates with several observable signals. Long requests with complex multi-step instructions tend to require more reasoning than short factual questions. Requests that ask the model to synthesize information from multiple domains, construct structured outputs (like code or JSON), or perform mathematical reasoning are systematically harder than requests that ask for straightforward retrieval. Requests involving ambiguous pronouns, implicit context, or domain-specific terminology challenge models that lack that context.

But the correlation is imperfect. A single-sentence question like "What is the integral of $e^x \sin(x)$?" is short but difficult. A paragraph-long request asking for a simple rewrite might be trivially easy. Length, vocabulary, and structural complexity are proxies for difficulty, not direct measurements of it. Any routing system built on proxies will make mistakes, and understanding where those mistakes cluster is essential for improving the system over time.

The most principled definition of difficulty is relative to the models available: a request is "hard" for a given model tier if that tier's output quality falls below an acceptable threshold. This makes difficulty a property of the request-model pair, not the request alone. The same question that stumps a 7B model may be trivially answered by a 70B model. This insight motivates the cascade architecture, where difficulty is evaluated dynamically rather than predicted statically.