LLM Inference Serving: Architecture, Routing & Auto-Scaling › Request Routing
Language AI HandbookAll chapters LLM Inference Serving: Architecture, Routing & Auto-Scaling › Request RoutingOne section · Sections in this chapter3 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When multiple model instances serve requests, a routing layer decides which instance handles each request. Effective request routing maximizes throughput while maintaining consistent latency. The routing layer must make rapid decisions with incomplete information, balancing immediate load distribution against longer-term efficiency considerations.
Think of request routing as air traffic control for your model fleet. Just as a controller assigns incoming flights to runways based on runway availability, weather, aircraft size, and priority, the router assigns incoming requests to model instances based on current load, request characteristics, priority level, and session history. A controller who assigns every plane to the nearest runway without checking whether it's occupied creates collisions; a router that sends every request to the same endpoint regardless of load creates a bottleneck just as severe.
The routing layer sees a stream of requests with varying characteristics: different prompt lengths, different expected output lengths, different priority levels, and different session histories. Making good routing decisions requires incorporating all this information into a fast decision that minimizes resource waste. Routing decisions that take longer than a few milliseconds add measurable overhead to every request, so the algorithms involved must be computationally lightweight even as they become semantically richer.
Production routing systems often combine multiple strategies in a hierarchy. A top-level policy might route requests to a specific model family based on capability requirements. A second level might balance load across instances of that model family using token-aware algorithms. A third level might check session affinity to enable KV cache reuse. This layered approach makes each individual routing decision simple while achieving sophisticated aggregate behavior.
Model Routing Strategies
Simple deployments route all requests to a single model version, treating the routing problem as pure load balancing. Production systems often need more sophisticated routing that considers request characteristics and business requirements.
Version-based routing directs requests to different model versions based on request metadata. This enables A/B testing by directing a fraction of traffic to a new model version, gradual rollouts that progressively shift traffic to updated models, and fallback to stable versions if new deployments exhibit problems. For example, a system might route 5% of traffic to a new model version while monitoring quality metrics before expanding the rollout.
Capability-based routing matches requests to appropriate models based on task complexity or domain. A lightweight model might handle simple queries like basic factual questions, while complex reasoning tasks route to larger, more capable models. This approach reduces cost without sacrificing quality where it matters: simple questions get fast, cheap answers while challenging problems receive appropriate computational resources.
Priority routing ensures high-priority requests (paying customers, necessary applications) get preferential treatment in terms of both queue position and endpoint selection. Lower-priority requests may queue longer or route to less powerful instances. This enables tiered service levels without requiring completely separate infrastructure for each tier.
import random
from dataclasses import dataclass
from typing import Dict, List, Optional
@dataclass
class ModelEndpoint:
"""Represents a model instance that can serve requests."""
endpoint_id: str
model_name: str
capacity: int # Max concurrent requests
current_load: int = 0
is_healthy: bool = True
@property
def available_capacity(self) -> int:
return self.capacity - self.current_load if self.is_healthy else 0
class ModelRouter:
"""Routes requests to appropriate model endpoints."""
def __init__(self):
self.endpoints: Dict[str, List[ModelEndpoint]] = {}
def register_endpoint(self, model_name: str, endpoint: ModelEndpoint):
if model_name not in self.endpoints:
self.endpoints[model_name] = []
self.endpoints[model_name].append(endpoint)
def route_request(
self, model_name: str, priority: int = 0
) -> Optional[ModelEndpoint]:
"""
Find the best endpoint for a request.
Higher priority requests can preempt lower priority ones.
"""
candidates = self.endpoints.get(model_name, [])
# Filter to healthy endpoints with capacity
available = [e for e in candidates if e.available_capacity > 0]
if not available:
return None
# For high priority, pick least loaded
# For low priority, pick randomly among available
if priority > 5:
return min(available, key=lambda e: e.current_load)
else:
return random.choice(available)# Demonstrate routing behavior
router = ModelRouter()
# Register multiple endpoints for the same model
for i in range(3):
endpoint = ModelEndpoint(
endpoint_id=f"llama-{i}",
model_name="llama-7b",
capacity=100,
current_load=random.randint(20, 80),
)
router.register_endpoint("llama-7b", endpoint)
# Route some requests at different priorities
routing_results = []
for priority in [1, 5, 8]:
selected = router.route_request("llama-7b", priority=priority)
routing_results.append((priority, selected))Registered Endpoints: llama-0: load=60/100 llama-1: load=27/100 llama-2: load=21/100 Routing Decisions: Priority 1 -> llama-2 (load: 21) Priority 5 -> llama-1 (load: 27) Priority 8 -> llama-2 (load: 21)
The router considers both capacity and priority when making routing decisions. High-priority requests route to the least-loaded endpoint to minimize queuing time and ensure fast response. Lower-priority requests distribute randomly among available endpoints to prevent creating hotspots while still avoiding overloaded instances. This differentiated treatment allows the system to provide better service to priority traffic without completely starving lower-priority requests.
Health Checks and Failover
Effective routing depends on accurate health information about each endpoint. If the router sends requests to an unhealthy endpoint, those requests will fail or experience long delays. Servers implement multiple health check levels to detect different kinds of problems.
Think of health checks as the monitoring equivalent of a doctor's examination at different levels of detail. A liveness check is like checking whether a patient is breathing: a minimal test to confirm the system is alive. A readiness check is like checking whether a patient can walk: verifying that the system can do what it's supposed to do. A deep health check is like a full physical: running complete diagnostics to catch subtle problems before they become emergencies. Each check level involves progressively more work and catches progressively more subtle issues.
The latency of health checks matters. Checks that take hundreds of milliseconds delay routing decisions and consume resources on the endpoints being checked. Deep health checks that run test inferences are especially expensive because they consume the same GPU resources needed for real requests. Production systems typically run liveness and readiness checks every few seconds, while deep health checks run every minute or on a triggered basis when shallow checks detect potential problems.
-
Liveness checks verify the process is running and responding to basic probes. A hanging process fails liveness checks even if it hasn't technically crashed. These checks detect catastrophic failures quickly.
-
Readiness checks verify the server can accept new requests. A server in the process of loading a model passes liveness (the process is running) but fails readiness (it cannot yet serve requests). This distinction prevents routing traffic to instances that are starting up.
-
Deep health checks verify end-to-end functionality by running test inferences. These checks catch subtle issues like corrupted model weights, CUDA driver problems, or memory leaks that wouldn't be detected by simpler checks. The trade-off is that deep checks consume GPU resources and take longer to complete.
import time
from dataclasses import dataclass
from typing import Callable, Dict, Optional
@dataclass
class HealthStatus:
is_live: bool = True
is_ready: bool = True
last_check: float = 0.0
consecutive_failures: int = 0
error_message: Optional[str] = None
class HealthChecker:
"""Monitors endpoint health with configurable checks."""
def __init__(
self,
check_interval: float = 10.0,
failure_threshold: int = 3,
recovery_threshold: int = 2,
):
self.check_interval = check_interval
self.failure_threshold = failure_threshold
self.recovery_threshold = recovery_threshold
self.endpoint_health: Dict[str, HealthStatus] = {}
def check_endpoint(
self,
endpoint_id: str,
liveness_fn: Callable[[], bool],
readiness_fn: Callable[[], bool],
) -> HealthStatus:
"""Perform health check and update status."""
status = self.endpoint_health.get(endpoint_id, HealthStatus())
try:
status.is_live = liveness_fn()
status.is_ready = readiness_fn() if status.is_live else False
if status.is_live and status.is_ready:
status.consecutive_failures = 0
status.error_message = None
else:
status.consecutive_failures += 1
except Exception as e:
status.is_live = False
status.is_ready = False
status.consecutive_failures += 1
status.error_message = str(e)
status.last_check = time.time()
self.endpoint_health[endpoint_id] = status
return status
def should_remove_from_rotation(self, endpoint_id: str) -> bool:
"""Check if endpoint should be removed due to failures."""
status = self.endpoint_health.get(endpoint_id)
if status is None:
return False
return status.consecutive_failures >= self.failure_threshold# Simulate health checking
checker = HealthChecker(failure_threshold=3)
# Simulate checks for different endpoint states
scenarios = [
("healthy-endpoint", lambda: True, lambda: True),
("loading-endpoint", lambda: True, lambda: False),
("failing-endpoint", lambda: False, lambda: False),
]
check_results = []
for endpoint_id, live_fn, ready_fn in scenarios:
status = checker.check_endpoint(endpoint_id, live_fn, ready_fn)
check_results.append((endpoint_id, status))Health Check Results:
healthy-endpoint:
Live: True, Ready: True
Failures: 0
loading-endpoint:
Live: True, Ready: False
Failures: 1
failing-endpoint:
Live: False, Ready: False
Failures: 1Health checks enable automatic failover without human intervention. When an endpoint exceeds the failure threshold (typically 3 consecutive failures), the router removes it from rotation and stops sending new requests. This prevents user-facing errors from accumulating while the underlying issue is investigated. After the underlying issue resolves, consecutive successful checks restore the endpoint to service. The recovery threshold requires sustained health before the endpoint receives traffic again, preventing oscillation between healthy and unhealthy states.