Model Routing › Routing Strategies

Routing strategies span a wide range, from completely static rules to fully learned systems that adapt to incoming data. The right choice depends on your system's complexity, latency budget, and how much signal you have about request difficulty.

The simplest routing strategy uses deterministic rules to partition traffic. These rules operate on observable request features without any machine learning:

• Request length: Short requests (under 50 tokens) go to the small model; long requests go to the large model.
• Keyword matching: Requests containing technical keywords from a domain-specific vocabulary are routed to a specialized model.
• User tier: Premium users get the larger model; free-tier users get the smaller one.
• Time of day: Route to cheaper models during peak hours to control costs; use the better model during off-peak periods.
• Explicit task type: Requests beginning with "summarize" or "translate" may be handled reliably by a smaller model, while requests asking for "explain step by step" or "write code to" might be escalated automatically.

Rule-based routing is easy to implement, inspect, and debug. There are no training data requirements and no risk of distributional shift in the router itself. The rules are transparent: every routing decision can be traced to a specific condition, which matters in regulated industries or when debugging quality regressions.

The downside is brittleness. Rules cannot capture the full complexity of what makes a request hard. A short request can require sophisticated reasoning. A long request might be a simple document with repetitive structure. Rules drawn from proxy features like length or keywords will misclassify these cases systematically. As your product evolves, you accumulate exceptions, special cases, and workarounds that make the rule set increasingly fragile. A request that was easy six months ago may require the large model today because the task has grown more demanding with your user base.

Rule-based routing works best as a starting point, not a final destination. It establishes a baseline from which you can measure improvements as you gather data and build more sophisticated classifiers.

A more principled approach estimates task difficulty directly. Difficulty-based routers learn a mapping from request features to a difficulty score, then threshold that score to select a model tier.

The intuition is that request difficulty is often predictable from surface features: the type of question (factual vs. reasoning), the domain (general vs. specialized), the presence of multi-step instructions, the required output format, or historical patterns for similar queries.

A difficulty classifier can be trained on labeled data where each request has an associated quality label from each model tier. A request is "hard" if the small model fails but the large model succeeds. This creates a supervised learning problem: predict whether a given request will cause the small model to fail.

The classifier can be as simple as a logistic regression over a small feature vector, or as complex as a fine-tuned language model that reads the request and outputs a difficulty score. In practice, the classifier needs to be much faster than the models it routes between, which pushes toward lightweight approaches. A gradient-boosted tree trained on a hundred features can make a routing decision in under a millisecond. A fine-tuned 1B-parameter model might take 50 milliseconds, which is acceptable if you are routing between a 500ms small model and a 5,000ms large model, but not if you are routing between a 30ms and a 200ms model.

The challenge is acquiring labeled data. You can bootstrap it by running a sample of traffic through all model tiers simultaneously, collecting outputs, and using human evaluation or an automated judge to compare quality. This is expensive initially but creates a feedback loop that improves the router over time. Each new labeled example makes the difficulty classifier more accurate, which in turn makes routing decisions better, which produces clearer signals for the next round of evaluation.

Cost-aware routing explicitly incorporates economic constraints into the routing decision. Rather than picking a model purely on predicted quality, it maximizes a utility function that weighs expected quality against expected cost.

For a request $r$, the routing decision selects model $m^*$ from a set of available models $\mathcal{M}$:

$$m^* = \arg\max_{m \in \mathcal{M}} \left[ Q(r, m) - \lambda \cdot C(r, m) \right]$$

where:

• $Q(r, m)$: the expected quality score for request $r$ handled by model $m$
• $C(r, m)$: the expected cost of routing request $r$ to model $m$
• $\lambda$: a cost-quality tradeoff parameter that controls how aggressively to prefer cheaper models

This formulation makes the tradeoff explicit. A high $\lambda$ routes aggressively to cheaper models; a low $\lambda$ prioritizes quality. In practice, $\lambda$ can be tuned globally or adjusted dynamically based on remaining budget for the billing period.

The elegance of this formulation is that it handles multi-tier decisions naturally. With three or more model tiers, you do not need separate rules for each pair. The optimizer considers all available models and selects whichever maximizes the utility function. You can add a new model tier without restructuring the entire routing logic, simply by adding it to $\mathcal{M}$ with its cost and quality estimates.

One important subtlety is that $Q(r, m)$ must be estimated before routing, not measured after. You do not know the actual quality of model $m$'s response to request $r$ until you generate it. In practice, $Q(r, m)$ is a predicted expected quality, estimated from features of $r$ and historical quality data for model $m$. This prediction introduces uncertainty: if the quality estimate is noisy, the routing decision may be suboptimal. Systems that track prediction confidence can handle this by routing to the larger model when quality estimates are uncertain, essentially treating uncertainty as a signal to be cautious.

Dynamic $\lambda$ adjustment is an important practical feature. Early in a billing cycle, when budget is ample, you might set $\lambda$ low to prioritize quality. As you approach the end of the cycle, $\lambda$ increases, pushing more traffic to cheaper models. This gives operators direct control over the cost-quality tradeoff as a function of budget state.

Embedding-based routers represent each incoming request as a vector in a shared embedding space and use similarity to previously seen requests (with known routing outcomes) to decide where to route the new request.

The approach works as follows. When a request arrives, you compute its embedding using a lightweight encoder. You then look up its nearest neighbors in a database of historical requests with known outcomes (quality scores from each model). If similar requests were handled well by the small model, you route there. If similar requests required the large model, you escalate.

This approach captures semantic similarity that keyword rules miss. A new question about Riemannian geometry that you have never seen before may share semantic space with previous calculus questions that the small model handled poorly. Embedding-based routing can generalize from a small number of labeled examples to handle novel requests gracefully because it uses continuous vector similarity rather than discrete rule matching.

The tradeoff is latency: the embedding computation and nearest-neighbor lookup add overhead on every request. For systems with strict latency budgets, this overhead may be prohibitive unless the lookup is highly optimized. Approximate nearest-neighbor algorithms like FAISS or HNSW make the lookup fast enough for practical use: retrieving the top-k neighbors from a database of a million vectors takes under 5 milliseconds on modern hardware.

Another limitation is the cold start problem. Embedding-based routing requires a sufficiently large and diverse database of labeled examples to generalize well. A sparse database with only a few dozen examples will not cover the request distribution adequately, and the nearest-neighbor matches will be unreliable. This makes embedding-based routing more suitable as a mature-phase routing strategy, once you have accumulated substantial labeled data, rather than a system you deploy on day one.

A learned routing classifier trains a lightweight model specifically to predict which backend model should handle a given request. This is typically a binary or multi-class classifier:

• Binary: "Should this request go to the small model or the large model?"
• Multi-class: "Which of these five model tiers should handle this request?"

The classifier inputs are typically lightweight features derived from the request: token counts, perplexity under a small language model, POS tag distributions, presence of domain keywords, or embeddings from a small encoder. The output is a routing decision.

The key design principle is that the classifier must be much cheaper than the models it routes between. A classifier that takes 50ms to make a routing decision saves nothing if it prevents a 30ms call to the small model. Practical routing classifiers are typically small linear models, gradient-boosted trees, or compact neural networks with inference times under 5ms.

Training data for the classifier comes from historical routing outcomes. Each training example consists of a feature vector derived from the request and a label indicating which model tier produced acceptable quality. This creates a natural label definition: if the small model's quality score exceeded threshold $\tau_{quality}$ on a given request, label it "small"; otherwise label it "large." The classifier learns to predict this label from request features without ever seeing the model responses at routing time.

One practical consideration is class imbalance. If 80% of your requests are easy (small model is sufficient), the classifier will see four times as many positive examples as negative ones. Standard techniques for handling class imbalance apply: reweighting the loss function, oversampling the minority class, or using calibrated probability outputs with a decision threshold adjusted to reflect the desired escalation rate.