Model Routing › Routing in Practice

Understanding routing strategies conceptually is one thing; implementing them reliably in a production system requires careful attention to engineering details.

The quality of a learned router depends heavily on the features it uses to represent incoming requests. Useful features include:

• Length features: Token count, word count, sentence count, average sentence length. These are cheap to compute and carry useful signal, even if imperfect.
• Linguistic features: Part-of-speech distributions, presence of questions, imperative phrases, or list structures. Requests with multiple embedded questions tend to be harder than single-question requests.
• Domain signals: TF-IDF scores against domain-specific corpora, keyword presence from domain vocabularies. A request containing dense mathematical notation is likely harder than a request about everyday topics.
• Complexity proxies: Perplexity of the request under a small language model (higher perplexity often correlates with harder requests), sentence embedding norms, or vocabulary richness measured by type-token ratio.
• Historical context: Whether similar requests in recent history were handled well by the small model, user-specific routing history.
• Structural features: Whether the request contains code, lists, mathematical expressions, or URLs. Requests that ask for code generation consistently require more capability than plain text requests.

Importantly, all these features must be computed quickly. If feature extraction takes 50ms, it dominates the latency budget for requests that are correctly routed to the 30ms small model. Feature engineering is a constant balance between predictive power and computational cost.

One practical technique is a two-level feature hierarchy. A first level of very cheap features (token count, simple keyword flags, character-level statistics) makes a fast initial routing decision for the clear-cut cases: obviously short and simple requests go to the small model, obviously long and complex requests go to the large model. A second level of more expensive features (perplexity, embedding similarity) is applied only to the ambiguous middle cases. This tiered approach concentrates the expensive feature computation where it matters most.

Every routing system has a latency budget. The routing decision itself adds overhead to every request. For requests that end up at the small model, the router adds latency. For requests that end up at the large model, the routing overhead is negligible relative to the large model's response time.

The break-even point for a router is when the overhead it adds to fast requests is justified by the savings from correctly avoiding the slow model. If your small model takes 50ms and your large model takes 500ms, and your router takes 10ms, then:

• Fast requests: 50ms + 10ms = 60ms (20% overhead)
• Slow requests (without router): 500ms
• Slow requests (with router, correctly routed to large): 500ms + 10ms = 510ms (2% overhead)

The routing overhead is acceptable as long as the router correctly avoids the large model often enough to compensate. If the router adds 10ms but only routes 5% of requests away from the large model, it is not worth it.

When routing overhead is a concern, asynchronous precomputation can help. For user-facing applications with a chat interface, you can begin computing routing features (and even the routing decision) while the user is still typing, before the request is submitted. By the time the request arrives, the routing decision is already made. This effectively eliminates routing latency from the user-perceived path.

Another latency optimization is request batching. If you aggregate multiple requests and route them as a batch, the per-request overhead of feature computation and classifier inference is amortized across the batch. Batch routing is most applicable in offline or near-real-time settings, not in interactive applications where each request must be routed immediately.

A learned routing classifier requires training data, but you do not have routing outcome labels before you deploy the system. This is the cold start problem.

Common bootstrapping approaches:

1. Shadow evaluation: Run both models on all requests without using their outputs to make decisions. Collect quality labels from human evaluation or an automated judge. Use this to build the initial training set for the router.
2. Heuristic initialization: Deploy a rule-based router first. Use its routing decisions (imperfect as they are) as initial labels, then iterate as you collect better feedback.
3. Offline datasets: If you have historical request data, run it through both models offline and use the results to train the initial router.
4. Synthetic data: Generate labeled examples by constructing requests of known difficulty (simple retrieval questions, multi-step reasoning problems) and evaluating model quality on them. Synthetic data is imperfect but can bootstrap a router until real traffic data accumulates.

The cold start problem means that routing system quality improves over time as more labeled data accumulates. Designing the system to continuously collect routing outcome feedback is as important as the initial routing design. This requires an explicit feedback collection pipeline: after each response, log the model tier used, the request features, and some quality signal (user rating, task completion flag, LLM judge score). This logged data is the training set for future router improvements.

A static routing classifier degrades over time as request distributions shift. A user base that starts with simple questions may evolve to ask more complex ones. New product features may introduce request types the router has never seen. Domain terminology changes.

Online learning addresses this by continuously updating the routing classifier as new labeled data arrives. In its simplest form, this means periodically retraining the classifier on a sliding window of recent requests. More sophisticated systems use online gradient descent to update model weights in real time.

However, online learning introduces its own risks. A classifier that learns too quickly from recent data can overfit to short-term distribution shifts, becoming unstable. A sudden spike in hard requests (say, due to a marketing campaign that attracts expert users) should not permanently shift the router toward always routing to the large model. Careful learning rate selection and regularization are needed to balance adaptability against stability.

An intermediate approach is to retrain the classifier on a schedule (weekly or monthly) using a rolling window of labeled data, rather than updating continuously. This provides adaptation to long-term distribution shifts without the instability of real-time updates. The retrained classifier is validated on a held-out set before deployment to catch regressions.

A production routing system needs dedicated monitoring to remain healthy. The key signals to track are:

• Escalation rate trend: If escalation rate is steadily increasing over weeks, it suggests the small model is increasingly struggling with incoming requests. This may indicate distributional shift, model degradation, or a product change that is attracting harder queries.
• Per-tier quality distribution: Monitor quality scores separately for requests handled by the small model and the large model. If small model quality scores start declining without a corresponding change in escalation rate, the evaluator may be miscalibrated.
• Router accuracy on labeled samples: Periodically evaluate the router on a labeled holdout set. If accuracy declines, it is time to retrain.
• Cost per request over time: Sudden changes in average cost signal routing behavior changes that may not be intended.
• Latency by tier: Track latency distributions separately for small-model and large-model requests. This helps distinguish between routing overhead (affects all requests) and model latency changes (affects only one tier).

Good observability makes the difference between a routing system you trust and one you fear. Without it, you will discover problems from user complaints rather than from metrics dashboards.