Auto-scaling › Scaling Challenges Unique to LLMs

LLMs introduce scaling complications that do not appear in conventional web services. Understanding these challenges is necessary for building a scaling system that works in production. Many teams that have successfully auto-scaled conventional services discover that their existing tooling and intuitions need significant revision when applied to LLM workloads.

Transformer inference maintains a key-value cache for each active request. During the decode phase of a long-context conversation, the KV cache for a single request may contain hundreds of megabytes of state representing the model's attention computations over the entire conversation history. This state lives in GPU memory on the specific instance serving that request.

During auto-scaling decisions, it is tempting to simply terminate an instance when scaling in. But if that instance is currently serving long-context conversations, terminating it abruptly drops those requests and requires clients to retry from scratch. For a chatbot application where users are in the middle of multi-turn conversations, sudden termination means the conversation history is lost. The user experience is equivalent to a sudden crash.

There are two common mitigations. Graceful draining stops routing new requests to an instance that is marked for removal, then waits for all active requests to complete before terminating it. The instance continues serving its in-flight requests until they finish, then shuts down cleanly. For chat sessions, this can work well as long as session lengths are bounded. The risk is that very long-running sessions can prevent instances from draining for extended periods, tying up capacity that the policy has already decided to reclaim.

The second mitigation is KV cache migration: serializing the in-progress KV cache state and sending it to a remaining instance before terminating the source. This is technically complex: the KV cache must be serialized from GPU memory to CPU memory, transmitted over the network, and deserialized back into GPU memory on the target instance, all while the client connection is maintained. This requires specialized infrastructure and significantly increases the engineering complexity of the serving stack. However, it eliminates the request-loss problem entirely and allows scale-in without service disruption even for long-running conversations.

Prefix caching, discussed in Chapter 45.7, interacts with graceful draining in an important way. If the draining instance has cached the KV computations for a frequently used system prompt, allowing the instance to terminate means losing that cache. The receiving instances may experience a temporary performance degradation as they rebuild the cache from scratch.

As noted earlier, loading a large model into GPU memory is slow. On cloud platforms that provision bare-metal GPU instances on demand, there is also VM startup time on top of model loading. The combined cold start latency for a 70B parameter model can exceed three minutes in the worst case.

Practical production systems address this through pre-warmed instance pools: a small number of instances are kept running but unloaded (or loaded but idle), ready to start serving almost immediately when a scale-out event is triggered. The pool size is calibrated to absorb expected demand spikes without incurring too much idle cost. A typical approach is to maintain enough warm instances to handle the largest expected traffic ramp that reactive scaling alone cannot handle within the acceptable queue depth limit. If traffic can double in 2 minutes and instances take 3 minutes to start, you need a warm pool large enough to absorb that doubling without exceeding queue depth limits.

Container image caching is another critical optimization. If the model weights are baked into the container image and the image is cached on the host's local storage, loading time reduces to a local disk read rather than a network download from object storage. NVMe SSD storage on modern cloud instances provides read speeds of 5-10 GB/s, which is 5-10x faster than typical S3 or GCS download speeds. For a 14 GB 7B model in FP16, the difference is between roughly 1.5 seconds (local NVMe) and 14 seconds (network download).

Model weight snapshots extend this idea: instead of loading the model from scratch, you save a GPU memory snapshot after the model is loaded and warm, then restore from the snapshot for subsequent starts. NVIDIA's CUDA Driver API supports capturing and restoring GPU memory state, and some serving frameworks have built support for this pattern. The effective startup time then becomes the time to restore a memory snapshot, which can be under a second for models that fit in a single GPU.

Some production deployments use Kubernetes with custom admission controllers to schedule new model serving pods on nodes that already have the model weights cached on local disk. This turns model weight locality into a scheduling constraint, making sure that scale-out pods start on nodes where the startup will be fast.

LLM serving infrastructure is rarely homogeneous in large-scale deployments. You might have a base fleet of smaller GPU instances (A10G with 24 GB VRAM) supplemented by larger instances (A100 or H100 with 80 GB VRAM) for long-context requests that require more memory. A routing layer directs requests to the appropriate instance tier based on request characteristics.

Scaling policies need to be aware of instance types and must scale each tier independently. The scale-out trigger for the A10G tier (running out of VRAM due to long-context requests) is different from the trigger for the A100 tier (compute saturation due to high throughput). Applying a single fleet-wide utilization metric can lead to counterintuitive results: if A10Gs are 90% utilized and A100s are 30% utilized, a naive fleet-average of 60% might not trigger any scaling action, even though the A10G tier is under severe strain.

Scale-in decisions in heterogeneous fleets are also more complex. Terminating an A100 while keeping A10Gs running might not be correct if the remaining A10Gs cannot handle long-context requests. The scaling policy must account for both the overall fleet count and the minimum number of each instance type needed to serve all request types.

In Kubernetes, this is typically handled through node pools and node affinity rules. Each instance type is its own node pool with its own Horizontal Pod Autoscaler and its own scaling targets. Requests are routed to the appropriate pod pool based on a sidecar proxy or a custom admission webhook that inspects request headers.

For conventional web services, requests are roughly similar in cost: serving a 1 KB JSON response takes nearly the same compute as serving a 2 KB JSON response. For LLM services, request costs vary by orders of magnitude. A request to generate a 10-token response is perhaps 100x cheaper than a request to generate a 1,000-token response of the same model.

This variability means that request count is a poor unit of capacity planning. Two scenarios with identical request rates can have dramatically different compute requirements: 1,000 short requests per minute (average output: 50 tokens) and 1,000 long requests per minute (average output: 500 tokens) would require 10x different capacity. If your scaling metric is request rate or queue depth in terms of request count, you cannot distinguish these two scenarios.

The solution is token-based capacity planning. Instead of measuring requests per second, measure input tokens per second and output tokens per second separately. Each has a different cost profile: input tokens are processed in parallel during the prefill phase and are relatively cheap; output tokens are generated one at a time during the decode phase and are relatively expensive. A complete capacity model expresses the serving cost as a weighted sum of input token rate and output token rate, with the weights derived from profiling the model on your hardware.

Monitoring token rates also makes it possible to detect changes in request characteristics over time. If users start submitting much longer prompts without a corresponding increase in request count, the total token throughput will rise even though the request rate stays flat. A purely request-based monitoring system would miss this, while a token-based system would detect the increased load and trigger scaling before queues start growing.

When auto-scaling cannot respond fast enough to a sudden spike, traffic shaping becomes necessary. Rate limiting caps the number of requests accepted per time window per user or per API key. Request queuing with timeouts allows brief overload to be absorbed smoothly as instances come online. Priority queues let high-value requests jump ahead during constrained periods.

Rate limiting is typically implemented at the API gateway layer, before requests even reach the model serving infrastructure. When a user exceeds their rate limit, the API gateway returns a 429 Too Many Requests response with a Retry-After header indicating when the user can try again. This protects the serving fleet from individual users consuming disproportionate capacity during traffic spikes.

Request queuing with backpressure works differently: instead of rejecting requests that exceed current capacity, they are accepted into a queue and served in order as capacity becomes available. This is appropriate when clients can tolerate some delay and when the requests themselves are valuable enough to be worth queuing. The risk is queue depth growing unbounded during sustained overload. Queue depth limits and timeout policies ensure that requests that have waited too long are rejected rather than served stale.

Priority queues add a dimension of request importance. In a multi-tier pricing model, paid users might have higher priority than free-tier users. Time-sensitive requests (like real-time completions) might have higher priority than batch requests. During constrained periods, the priority queue ensures that high-value requests are served first, even if lower-priority requests must wait or be dropped.

Traffic shaping is not a substitute for good auto-scaling, but it is an essential complement. It prevents a short burst from causing cascading failures while the autoscaler catches up. The next chapter on caching explores how cached completions can further reduce the effective load that reaches the scaling layer.

Modern LLM serving frameworks like vLLM and TGI use continuous batching to maximize GPU utilization. In continuous batching, newly arrived requests are added to the current batch as soon as a slot opens up (when an existing request finishes generating a token), rather than waiting for an entire batch to complete before starting a new one. This dramatically improves throughput compared to static batching.

Continuous batching interacts with auto-scaling in subtle ways. When the serving engine is under-loaded, continuous batching may produce small batches (2-4 requests), which are GPU-inefficient but fast per request. When the serving engine is fully loaded, continuous batching produces large batches (16-32 requests), which are GPU-efficient but slower per request due to increased memory bandwidth contention.

This means that a single GPU utilization number does not fully capture the serving state. An instance running at 60% GPU utilization with a batch size of 4 is in a very different state from an instance running at 60% utilization with a batch size of 16. The first instance has room to absorb more requests without latency degradation; the second is already near its latency limit. Scaling policies that are aware of batch size can make better decisions, but batch size monitoring requires integration with the serving framework's internal metrics.

vLLM exports a rich set of metrics via its Prometheus endpoint, including current batch size, KV cache utilization, queue depth in tokens (not just requests), and per-request waiting times. Building auto-scaling policies on top of these framework-native metrics gives much more precise control than relying solely on hardware-level GPU utilization.