LLM Inference Serving: Architecture, Routing & Auto-Scaling › Summary

This chapter covered the infrastructure layer that turns trained language models into production services.

Inference server architecture comprises specialized components including request handlers, schedulers, model executors, and memory managers. Modern servers like vLLM and TGI implement sophisticated features like PagedAttention and continuous batching that make them dramatically more efficient than simple model serving. The request lifecycle spans queuing, tokenization, prefill, decode, and completion, with each stage presenting distinct optimization opportunities.

Request routing determines which model instance handles each request, supporting version-based routing for A/B testing, capability-based routing for cost optimization, and priority routing for SLA differentiation. Health checks ensure routing decisions reflect actual endpoint availability and enable automatic failover without human intervention.

Load balancing for LLMs requires token-aware approaches that account for variable request costs. Traditional least-connections balancing fails to capture the difference between short and long generations. Session affinity enables KV cache reuse but conflicts with optimal load distribution, and real systems balance these two objectives based on traffic patterns and cache efficiency gains.

Auto-scaling adjusts capacity based on metrics like queue depth, latency percentiles, and GPU utilization. Cold start latency for LLM instances (often 30-120 seconds) necessitates warm pools and predictive scaling to maintain responsiveness during demand spikes. The asymmetric cooldowns for scale-up and scale-down prevent capacity oscillation while allowing a rapid response to demand increases.

Latency optimization requires understanding the distinct prefill (compute-bound) and decode (memory-bound) phases. Time-to-first-token matters for interactive applications, while total latency matters for batch processing. Bottleneck analysis reveals whether optimization should target prefill speed, decode speed, or queue reduction, and investing in the wrong dimension wastes effort.

Monitoring tracks request-level metrics (TTFT, latency), system-level metrics (GPU utilization, queue depth), and business metrics (tokens per second, cost). SLO-based alerting with warning thresholds provides early warning before service degradation affects users.

These serving concepts build on the inference optimization techniques from earlier chapters in this part. Together, they enable deploying language models that scale from prototype to production, handling thousands of concurrent users while meeting latency and cost targets.