Auto-scaling › Auto-Scaling

When an LLM-powered service launches, its traffic is rarely constant. A product release can multiply request volume tenfold overnight. A scheduled batch job drains resources while users sleep. A viral social post can generate a traffic spike before your on-call engineer even opens their laptop. Traditional approaches to this problem either over-provision capacity (wasting money during quiet periods) or under-provision it (dropping requests during peaks). Auto-scaling resolves this tension by continuously adjusting the number of serving instances in response to real demand.

This chapter covers the mechanics of auto-scaling in the context of LLM production systems. You will learn how to choose scaling metrics that predict load accurately, understand the architectural tradeoff between scaling up (more powerful machines) and scaling out (more machines), implement reactive and proactive scaling policies, and handle the failure modes that are unique to large model deployments. The chapter builds on the monitoring concepts from Chapter 45.5 and on the throughput and latency ideas from Chapters 45.4 and 45.3.

Auto-scaling balances systems engineering constraints against operating costs. Every running GPU instance costs money. Every request that fails due to insufficient capacity costs user trust. A well-tuned scaling system keeps these two costs balanced: just enough capacity, at just the right time. Getting this balance wrong in either direction has real consequences. Too conservative, and your service becomes unreliable during peaks. Too aggressive, and your infrastructure bill grows faster than your revenue. The mechanics of auto-scaling seem simple on the surface, but the details matter enormously in production, especially for LLM workloads where the cost per instance is high and the startup time for new instances is measured in seconds to minutes rather than milliseconds.

The history of auto-scaling in cloud computing stretches back to the early days of AWS Elastic Compute Cloud, when teams first realized that manually managing instance counts was a full-time job that still left services vulnerable to sudden demand changes. The solutions that emerged for conventional web services work reasonably well when request processing time is measured in milliseconds and container startup completes in a few seconds. LLM serving breaks both of those assumptions. A single LLM inference request may take seconds to complete. A new LLM serving instance may take a minute or more to become ready. These differences require rethinking every part of the auto-scaling stack, from metric selection to policy design to instance lifecycle management.