Auto-scaling › Summary

Auto-scaling matches LLM serving capacity to demand automatically, reducing both over-provisioning cost and under-provisioning failures. The key takeaways from this chapter are:

• Metric selection determines everything: Choose metrics that directly reflect user-visible impact (TTFT, queue depth, token throughput) rather than infrastructure proxies alone (CPU load). Composite metrics reduce sensitivity to any single signal and provide a more complete picture of serving health.
• Scale-out is the primary strategy: Add more instances rather than larger instances. Horizontal scaling is incremental, resilient, and does not require service interruptions. Vertical scaling is a pre-deployment capacity planning decision, not a runtime scaling mechanism.
• Policies differ in responsiveness and stability: Threshold-based scaling is simple but prone to flapping. Target tracking is more stable, mathematically principled, and widely supported by cloud platforms. Step scaling handles sudden spikes aggressively by adding multiple instances at once when utilization is critically high.
• Proactive and reactive complement each other: Scheduled scaling handles predictable daily and weekly patterns; reactive policies handle deviations. Predictive scaling can further reduce cold-start gaps for traffic with learnable patterns, but must always be backstopped by reactive policies for anomalies.
• Cold start latency is the central engineering constraint for LLMs: Startup times measured in tens of seconds to minutes force a combination of pre-warmed pools, fast model loading via local caching and memory snapshots, and traffic shaping to handle spikes gracefully. Reducing startup latency directly improves the service's ability to respond to sudden demand changes.
• LLM-specific challenges require dedicated solutions: KV cache state makes instance termination more complex than in stateless services. Heterogeneous hardware fleets require per-tier scaling policies. Token-based capacity planning gives more accurate capacity estimates than request-based metrics. Continuous batching dynamics affect the relationship between utilization and latency in non-obvious ways.
• Scaling bounds and monitoring reliability are as important as the policy itself: A correctly implemented target tracking policy will fail silently if the metric it tracks is unreliable, or if the scaling bounds prevent it from reaching the desired instance count. Operational discipline around reviewing and updating configuration is a prerequisite for auto-scaling to work in practice.

In the next chapter on caching, we will see how returning cached completions for repeated prompts can dramatically reduce the effective request rate reaching the scaling layer, making auto-scaling cheaper and more stable.