Auto-Scaling LLMs: Metrics, Policies, Production Strategies

When an LLM-powered service launches, its traffic is rarely constant. A product release can multiply request volume tenfold overnight. A scheduled batch job drains resources while users sleep. A viral social post can generate a traffic spike before your on-call engineer even opens their laptop. Traditional approaches to this problem either over-provision capacity (wasting money during quiet periods) or under-provision it (dropping requests during peaks). Auto-scaling resolves this tension by continuously adjusting the number of serving instances in response to real demand.

This chapter covers the mechanics of auto-scaling in the context of LLM production systems. You will learn how to choose scaling metrics that predict load accurately, understand the architectural tradeoff between scaling up (more powerful machines) and scaling out (more machines), implement reactive and proactive scaling policies, and handle the failure modes that are unique to large model deployments. The chapter builds on the monitoring concepts from Chapter 45.5 and on the throughput and latency ideas from Chapters 45.4 and 45.3.

Auto-scaling balances systems engineering constraints against operating costs. Every running GPU instance costs money. Every request that fails due to insufficient capacity costs user trust. A well-tuned scaling system keeps these two costs balanced: just enough capacity, at just the right time. Getting this balance wrong in either direction has real consequences. Too conservative, and your service becomes unreliable during peaks. Too aggressive, and your infrastructure bill grows faster than your revenue. The mechanics of auto-scaling seem simple on the surface, but the details matter enormously in production, especially for LLM workloads where the cost per instance is high and the startup time for new instances is measured in seconds to minutes rather than milliseconds.

The history of auto-scaling in cloud computing stretches back to the early days of AWS Elastic Compute Cloud, when teams first realized that manually managing instance counts was a full-time job that still left services vulnerable to sudden demand changes. The solutions that emerged for conventional web services work reasonably well when request processing time is measured in milliseconds and container startup completes in a few seconds. LLM serving breaks both of those assumptions. A single LLM inference request may take seconds to complete. A new LLM serving instance may take a minute or more to become ready. These differences require rethinking every part of the auto-scaling stack, from metric selection to policy design to instance lifecycle management.

Before any scaling action can be triggered, the system needs a signal that reliably indicates whether current capacity is adequate. Choosing the right metric is the most consequential decision in auto-scaling design. The metric must be sensitive enough to detect overload early, stable enough not to trigger spurious scale events, and available with low enough latency to allow a timely response.

The challenge is that common infrastructure metrics often lag the actual user experience. A CPU utilization spike tells you the machine is busy, but it does not tell you whether requests are succeeding or how long they are taking. For LLM services, the relationship between infrastructure metrics and user-visible outcomes is especially indirect because inference workloads are highly variable: a three-token completion and a 4,096-token completion are radically different in resource consumption, yet both count as one request. A metric that works well for a homogeneous workload (like serving static files) can give completely misleading signals when applied to highly variable inference workloads.

Understanding the failure modes of each metric is as important as understanding what the metric measures. A metric that causes your autoscaler to oscillate is often worse than no autoscaler at all, because it adds the overhead of repeated scale events on top of whatever traffic problem it was meant to solve.

GPU utilization measures the fraction of time the GPU compute units are actively executing kernels. A utilization value near 1.0 generally means the GPU is saturated and additional requests will queue. A value near 0.0 means the GPU is idle and instances can be removed.

GPU utilization has an intuitive appeal because it measures the thing you are paying for: GPU compute time. If GPUs are running at full utilization, you need more GPUs. If they are nearly idle, you are paying for capacity you do not need. This logic holds well for compute-bound workloads, but LLM inference is often memory-bandwidth-bound rather than compute-bound. During the decoding phase of autoregressive generation, each forward pass reads model weights and KV cache entries from GPU memory and performs a relatively small number of floating-point operations on them. The ratio of memory reads to compute operations is high enough that GPU compute units may sit waiting for memory even while utilization metrics suggest the GPU is underloaded.

The limitation of GPU utilization as a scaling metric is memory pressure. An LLM inference server can exhaust its GPU memory (VRAM) while GPU compute utilization remains moderate, because the KV cache from many concurrent long-context requests fills memory before compute becomes the bottleneck. If a server is hosting 20 concurrent long-context conversations, each with a 32,000-token context window and a 4-billion parameter model using FP16 weights, the KV cache alone may consume tens of gigabytes of VRAM. If you scale on utilization alone, you will discover out-of-memory crashes before your autoscaler triggers a scale-out.

The practical workaround is to monitor GPU memory utilization separately from GPU compute utilization. VRAM utilization climbing above 85-90% is a signal that new requests will soon start failing with out-of-memory errors, even if compute utilization is only at 60%. Good production monitoring systems treat both as independent scaling triggers.

Queue depth counts how many requests are waiting for a free serving slot. Unlike utilization, it directly measures user-visible impact. A queue of zero means all requests are being served immediately. A growing queue means demand exceeds supply.

Queue depth is a lagging indicator by design: the queue can only grow after demand has already exceeded capacity. The practical implication is that scaling on queue depth alone requires generous look-ahead in the scaling policy. You need to add instances fast enough that the queue drains before it accumulates too many waiting requests. If your instances take 90 seconds to start, and you only trigger a scale-out when the queue exceeds 10 requests, the queue could grow to 50 or 100 requests before the new instance is ready. By then, many of those waiting requests may have timed out on the client side.

Queued requests and queued tokens measure different things. A request asking the model to summarize a 10,000-word document will occupy a serving slot for much longer than a request asking for a one-sentence answer. If your queue depth metric counts requests without regard to their size, the apparent queue depth may look the same for two situations that have vastly different workloads. More sophisticated serving frameworks like vLLM track queued tokens rather than queued requests, which gives a more accurate picture of the work backlog.

Queue depth is most useful when combined with other metrics. On its own, it can mask the difference between a single very large request blocking the queue and ten small requests all waiting for separate reasons.

Time to first token measures how long a user waits from submitting a request until the first token of the response appears. It is one of the most direct proxies for perceived latency, since users notice the delay before a streaming response begins. In a streaming API where partial responses are delivered token by token, users experience TTFT as the initial "pause" before the text starts flowing. Even if the overall generation speed is fast, a long TTFT makes the service feel slow and unresponsive.

TTFT has two components: queuing time and prefill time. Queuing time is the delay introduced by waiting for a free serving slot. Prefill time is the cost of processing the input tokens through the model before generation can start. For long prompts, prefill itself can take hundreds of milliseconds even on a fast GPU. For the purposes of scaling, the queuing component is more actionable: you can reduce it by adding capacity. The prefill component is fixed by the model and hardware.

TTFT increases when the serving cluster is overloaded because requests spend more time waiting in queue before prefill begins. It is therefore a useful scaling metric: if p95 TTFT rises above a target threshold, the system needs more capacity. P95 (the 95th percentile) is typically more appropriate than the average or median here, because the worst 5% of users have the worst experience, and if that experience degrades significantly, you will hear about it in user complaints before it shows up in average metrics.

A practical complication is that TTFT varies significantly with prompt length. If your users submit prompts of wildly different lengths, the p95 TTFT will fluctuate for reasons unrelated to capacity. Services that want to use TTFT as a scaling metric often normalize it by prompt length, computing something like TTFT per 1,000 input tokens.

This metric normalizes throughput by the number of running instances. As we covered in Chapter 45.4, throughput optimization is central to LLM serving economics. Monitoring tokens-per-second-per-instance lets you detect when individual instances are underperforming (perhaps due to memory pressure) and when the fleet as a whole is approaching saturation.

When tokens-per-second-per-instance is declining while instance count is constant, it indicates that requests are becoming harder (longer context, more concurrent requests per instance) or that hardware is degrading (thermal throttling, ECC errors slowing memory access). This metric can therefore diagnose issues that pure utilization metrics miss. If utilization is flat but throughput per instance is declining, you have a quality problem rather than a capacity problem.

At the fleet level, tracking total tokens-per-second against an expected value derived from your peak-hour forecasts gives you a quick sanity check that the fleet is keeping up with demand. This is particularly useful for batch inference workloads where you have a known token budget to process within a deadline.

Inter-token latency, also called time between tokens or generation speed, measures how long the model takes to produce each successive token after the first. While TTFT captures the initial wait, ITL governs the streaming experience: a fast ITL makes responses feel fluid and fast, while a high ITL causes the streaming text to appear in slow, choppy bursts.

ITL is determined primarily by GPU compute and memory bandwidth, not by queue depth. It changes relatively little as a function of load, because each decoding step involves a forward pass through the model that takes roughly the same time regardless of how many requests are in the queue. The exception is continuous batching: when many requests are batched together during the decode phase, each individual request's ITL degrades somewhat because the batch is larger and the memory bandwidth is shared.

Including ITL in your scaling metrics helps detect when the serving system is over-batching: packing too many requests into each batch in an attempt to maximize throughput at the cost of per-request latency. A rising ITL alongside a stable TTFT can indicate this condition.

In practice, the most robust scaling signal is a composite of several indicators. A common approach is to define a scaling score that weighs GPU utilization, queue depth, and p95 TTFT simultaneously:

where:

• $\text{util}$ is the current GPU utilization averaged across the fleet
• $\text{util\_target}$ is the target utilization you want to operate at (often 0.75)
• $\text{queue}$ is the current request queue depth
• $\text{queue\_target}$ is the maximum acceptable queue depth
• $\text{TTFT}$ is the current p95 time to first token in milliseconds
• $\text{TTFT\_target}$ is the SLA target for time to first token
• $w_1, w_2, w_3$ are weights that sum to 1.0 and reflect how much each metric matters for your service

When the composite score exceeds 1.0, the system scales out. When it drops below a low threshold (often 0.5), the system scales in. The composite approach prevents any single metric from causing oscillation while remaining sensitive to the combination of signals that together indicate strain.

The weights $w_1, w_2, w_3$ encode your priorities. A service with a strict latency SLA might assign a high weight to TTFT: $w_3 = 0.6$, leaving the other two metrics as secondary confirmations. A batch processing service that cares more about throughput than latency might weight utilization most heavily: $w_1 = 0.6$. There is no universally correct weighting; it must be calibrated against your specific service requirements.

One important implementation detail: the metrics in the composite score should be smoothed over a rolling window before comparison. Raw per-second GPU utilization is noisy. A single second of 95% utilization during a burst of long prefills might not warrant a scale-out. A rolling average over 60-90 seconds gives a more stable signal. The window length is a tradeoff between responsiveness (shorter window) and stability (longer window).

Once you have decided that more capacity is needed, there are two fundamentally different ways to provide it: vertical scaling (scale-up) and horizontal scaling (scale-out). Understanding their tradeoffs is essential for designing a cost-effective and responsive LLM infrastructure.

Vertical scaling adds resources to an existing instance: a larger GPU, more VRAM, more CPU cores, more system RAM. For LLM inference, the most common form is moving from a smaller GPU (say, an A10G with 24 GB VRAM) to a larger one (an A100 with 80 GB VRAM). Alternatively, you might add a second GPU to a single node and run the model in tensor-parallel mode across both.

Vertical scaling is attractive when your bottleneck is model capacity rather than throughput. If a model does not fit in a single GPU's VRAM, scaling up is not optional. A 70B parameter model in FP16 requires approximately 140 GB of VRAM. A single A100 provides 80 GB, which is insufficient without quantization. To run the model on a single node, you need at least two A100s in tensor-parallel configuration, or one H100 with 80 GB (though even then quantization is typically required). This is a hard constraint that the scaling policy cannot work around: you must have enough VRAM per model replica before any scaling discussion is relevant.

Vertical scaling is also appealing for latency-sensitive workloads. A single A100 will generally produce tokens faster than a single A10G because its wider memory bandwidth lets the attention and feedforward computations execute more quickly. This matters for services where you care about individual request performance, not just aggregate throughput. Moving up from an A10G (600 GB/s memory bandwidth) to an H100 (3.35 TB/s) can reduce per-token generation time by a factor of five or more for bandwidth-bound workloads.

The drawbacks of vertical scaling are significant in an auto-scaling context. Most cloud providers do not support dynamic vertical scaling: you cannot add a GPU to a running instance without stopping and restarting it. This makes vertical scaling a pre-deployment decision rather than a runtime response to load changes. You choose the right instance type when you design the deployment, and you stick with it. Changing instance types in response to a live traffic spike is simply not feasible in most cloud environments.

Even when vertical scaling is technically possible (for example, by migrating workloads between instance types in a managed Kubernetes cluster), it is slow and disruptive. The economics are also often unfavorable: a single large instance is more expensive than two smaller instances with equivalent aggregate compute, partly due to hardware rarity and partly due to cloud provider pricing models that charge a premium for the largest instance types.

Horizontal scaling adds more instances running the same model. Each instance is a complete, independent copy of the serving process. A load balancer distributes incoming requests across all running instances.

Scale-out is the dominant strategy for LLM auto-scaling in production for several reasons. First, it is incremental: you can add one instance at a time, progressively increasing capacity without service interruption. Second, it is resilient: if one instance crashes or becomes unhealthy, the load balancer routes around it and the fleet continues serving. Third, it naturally supports geographic distribution: you can scale out in multiple regions independently, keeping each region's capacity proportional to its local traffic. Fourth, the economics are favorable: adding two A10G instances is typically cheaper than upgrading to one A100 for the same aggregate throughput.

The cost of scale-out is model startup time. Loading a large LLM from disk (or from object storage) into GPU memory is not instantaneous. A 70-billion parameter model in FP16 requires roughly 140 GB of data to transfer, which at 10 GB/s storage throughput takes around 14 seconds before the first request can be served. On cloud platforms that provision bare-metal GPU instances on demand, there is also VM startup time and container pull time on top of model loading. The combined cold start latency for a 70B parameter model can easily exceed three minutes in a worst-case cloud environment.

This startup latency is the central constraint that differentiates LLM auto-scaling from conventional web service auto-scaling. A conventional web application container starts in 2-5 seconds. An LLM serving instance starts in 30 seconds to 5 minutes. This difference changes everything about how policies must be designed, how far in advance proactive scaling must act, and how much idle capacity you must maintain as a buffer against sudden spikes.

Understanding this constraint also motivates a family of engineering optimizations aimed at reducing startup time. Container image caching ensures that the serving binary and library dependencies are already present on the host when a new instance is scheduled. Model weight caching on local NVMe storage reduces model loading from minutes (for a network download from object storage) to seconds (for a local disk read). Some systems maintain a pool of pre-initialized but idle serving processes, ready to accept traffic within seconds once they receive a routing signal.

The practical rule is to use scale-out as the primary scaling mechanism and treat scale-up as a pre-deployment decision. You choose the right instance type for your model's requirements when designing the deployment, then auto-scale by adding more of those instances in response to load. Vertical scaling during a live incident is rarely feasible given the disruption it causes.

This rule has an important corollary: instance type selection is a capacity planning decision, not a scaling decision. When you choose between running a 7B model on A10Gs or A100s, you are making a tradeoff between per-instance cost, per-instance throughput, and VRAM capacity. That decision should be made at deployment time based on your expected workload characteristics, not in response to live traffic.

The one exception is multi-GPU tensor parallelism: if a model requires 4 GPUs per instance (common for very large models), then your "instance" is a 4-GPU node, and horizontal scaling means adding more 4-GPU nodes. The principle remains the same. The unit of horizontal scaling is whatever constitutes a complete, self-contained serving replica, whether that is a single GPU, a multi-GPU node, or a multi-node cluster.

There is also a hybrid strategy that some large-scale deployments use: maintain a base fleet of smaller, cheaper instances for normal load, and use larger, more powerful instances as overflow capacity during peaks. A request router directs traffic to the appropriate tier based on request characteristics (for example, long-context requests might always go to the high-VRAM tier) or based on queue depth (overflow to the high-performance tier only when the base fleet is saturated). This requires more sophisticated routing logic but can be more cost-efficient than a homogeneous fleet.

A scaling policy defines the rules that translate metric observations into scaling actions. Reactive policies respond to what the system is currently experiencing. They are the foundation of any auto-scaling system and handle the majority of scaling events in most production deployments.

The core challenge in designing reactive policies is balancing responsiveness against stability. A policy that reacts instantly to every metric fluctuation will produce a system that constantly adds and removes instances, wasting the startup time of each instance it provisions and never reaching a steady state. A policy that waits too long to react will allow queue depth and latency to grow for extended periods before relief arrives. Good reactive policies are tuned to respond quickly to meaningful load changes while ignoring transient noise.

The simplest reactive policy scales out when a metric crosses an upper threshold and scales in when it crosses a lower threshold. For example: add one instance when GPU utilization exceeds 80%, remove one instance when it falls below 30%.

Threshold-based scaling is easy to understand and implement. Its weakness is oscillation: if the metric hovers near the threshold, the system may repeatedly add and remove instances, wasting the startup time of each instance it provisions. This is called "flapping." Flapping incurs constant startup costs without allowing the system to stabilize, and the repeated scale-in and scale-out events can cause service disruptions as the load balancer redistributes traffic each time.

Cooldown periods are the standard defense against flapping. A cooldown period is a mandatory waiting time after any scaling action before the next one is allowed. If scale-out takes 60 seconds to provision and stabilize a new instance, a 120-second cooldown prevents triggering another scale-out before the first one has had a chance to reduce the metric.

Separate cooldowns for scale-out and scale-in are an important refinement. Scale-out cooldowns are typically shorter because you want to respond quickly to load growth. Scale-in cooldowns are typically longer because premature scale-in wastes the startup cost of the instance that was just added. A common configuration might use a 90-second scale-out cooldown and a 300-second scale-in cooldown.

The hysteresis gap (the difference between the scale-out threshold and the scale-in threshold) also prevents flapping. In the example above, the 50-percentage-point gap between the 80% scale-out threshold and the 30% scale-in threshold means that a new instance, once added, must reduce utilization below the lower threshold before the system considers removing it again.

Step scaling generalizes threshold-based policies by defining multiple thresholds, each triggering a different number of instances. For example:

• Utilization between 80% and 90%: add 1 instance
• Utilization between 90% and 95%: add 2 instances
• Utilization above 95%: add 4 instances

This allows the system to respond more aggressively to severe overload while avoiding over-reaction to mild load increases. Step scaling is well-suited to traffic patterns with sudden, large spikes: if traffic doubles in 60 seconds, a step policy can add 4 instances at once instead of slowly adding them one at a time through repeated threshold crossings.

The step boundaries require tuning for each service. A poorly calibrated step policy with large steps can over-provision massively during a transient spike, wasting significant money before the scale-in cooldown allows the fleet to shrink. Step policies work best when you have well-understood traffic distributions and have tested the policy under realistic load scenarios.

For LLMs specifically, step scaling during the scale-out direction is often appropriate because the cost of adding instances is high (both in actual GPU cost and in the coordination overhead of redistributing load) and you want to avoid repeatedly adding one instance at a time during a rapid traffic ramp. Adding 4 instances at once when utilization hits 95% is often more efficient than the escalating sequence of threshold crossings that a simpler policy would generate.

Target tracking inverts the logic: instead of defining thresholds and actions, you define a target value for the metric and let the scaler continuously adjust instance count to maintain it. If the target is 70% GPU utilization and current utilization is 90%, the scaler computes how many additional instances are needed to bring average utilization back to 70%.

where:

• $\text{desired\_instances}$ is the computed number of instances needed
• $\text{current\_instances}$ is how many instances are running now
• $\text{current\_metric}$ is the current metric value (e.g., utilization = 0.90)
• $\text{target\_metric}$ is the desired metric value (e.g., 0.70)
• $\lceil \cdot \rceil$ is the ceiling function, since you cannot run fractional instances

This formula gives a proportional response: the further the current metric is from the target, the larger the adjustment. At moderate overload (say, 84% vs 70% target with 5 running instances), the formula suggests 6 instances. At severe overload (95% vs 70% target with 5 instances), it suggests 7 instances. The response scales continuously with the severity of the overload.

Target tracking is the most widely used policy in production because it naturally handles a wide range of load levels without requiring manual threshold tuning. AWS Auto Scaling Groups, Kubernetes Horizontal Pod Autoscaler, and Google Cloud Managed Instance Groups all offer target tracking as a first-class policy. You specify the target metric value and let the autoscaler do the arithmetic.

One subtlety of target tracking for LLMs: the formula above assumes a linear relationship between instance count and metric value. If adding one instance reduces average utilization from 90% to 78%, target tracking will work well. But LLM workloads sometimes exhibit non-linear relationships due to continuous batching. Adding more serving capacity can allow the serving engine to batch more requests together, which can reduce per-request throughput (because each request waits longer for a full batch) while increasing overall tokens-per-second. In these situations, target tracking on utilization alone may not converge to the expected operating point, and the composite metric approach becomes more important.

Reactive policies always trail demand by at least one metric collection cycle plus instance startup time. For LLMs with their substantial startup latency, even a 60-second delay between the metric crossing a threshold and a new instance becoming ready can mean hundreds of dropped requests. Proactive scaling eliminates the lag for predictable load patterns by adding capacity before demand arrives.

Proactive and reactive policies are complements, not alternatives. Proactive scaling handles the predictable patterns that repeat daily and weekly. Reactive scaling handles the deviations from those patterns: an unexpectedly busy Monday, a viral mention that drives a traffic spike during an otherwise quiet period, or a scheduled maintenance window that compresses traffic into a smaller time window.

Scheduled scaling sets a minimum instance count at specific times. If your LLM service handles enterprise customers who all log in between 9 AM and 10 AM in their local timezone, you can schedule a scale-out at 8:45 AM and a scale-in at 7 PM.

Scheduled scaling is reliable for predictable traffic patterns and does not require any metric measurement. Its weakness is that it does not adapt to anomalies. If a scheduled event is cancelled and traffic is unexpectedly low, the scaled-out fleet wastes money. If an unscheduled event creates traffic outside of the scheduled window, the schedule provides no help.

In practice, scheduled scaling is used as a floor for reactive scaling: you schedule a minimum capacity that handles expected load, and reactive policies handle deviations above and below it. The scheduled minimum prevents the reactive policy from scaling in too aggressively during a brief lull between expected traffic peaks.

Timezone management is a practical concern that becomes important at scale. An enterprise service with customers in multiple continents may have overlapping business-hours windows that create a rolling "wave" of scheduled peaks across a 24-hour period. Each timezone transition needs to be accounted for in the schedule, or the schedule quickly becomes incorrect during daylight saving time transitions.

Predictive scaling trains a forecasting model on historical traffic data to predict future load, then provisions instances in advance based on those predictions. The forecast horizon must be at least as long as the instance startup time to be useful. For a system where instances take 3 minutes to be ready, you need a forecast that is accurate 3 minutes into the future. For practical safety margin, you want a forecast horizon of 5-10 minutes.

For daily and weekly patterns, even simple forecasting models (ARIMA, exponential smoothing, or a basic neural network) can achieve enough accuracy to substantially reduce cold-start gaps. The key insight is that you do not need to predict the exact traffic level, only whether you need more capacity than you currently have. A conservative forecast that over-predicts load by 20% is often preferable to a precise forecast that is occasionally wrong in the under-prediction direction.

Cloud providers like AWS offer predictive scaling as a managed feature that analyzes historical CloudWatch metrics and generates hourly forecasts automatically. Google Cloud also offers predictive autoscaling based on historical load patterns. These managed services reduce the engineering burden of building and maintaining a custom forecasting pipeline.

The critical failure mode of predictive scaling is distribution shift: if a product launch, a marketing campaign, or a news event generates traffic unlike any historical pattern, the predictor will under-forecast. A model trained on two months of historical data has never seen the traffic pattern created by being mentioned on a major news site. Predictive scaling should always be combined with reactive policies as a safety net for unforeseen spikes. The combination is: predictive scaling ensures capacity is ready for the expected pattern, reactive scaling handles everything outside that pattern.

To make these concepts concrete, let us trace through a complete scaling decision for a hypothetical LLM service. The service runs a 7B parameter model on A10G instances (24 GB VRAM each). Current state: 4 instances running, 78% average GPU utilization, a request queue depth of 3, and p95 TTFT of 850 ms against a target of 500 ms.

The composite scaling score combines all three metrics. The weights here reflect a service that prioritizes latency above utilization: $w_1 = 0.4$ for utilization, $w_2 = 0.3$ for queue depth, and $w_3 = 0.3$ for TTFT, with targets of 0.70, 5 queued requests, and 500 ms respectively.

The score exceeds 1.0, so a scale-out is triggered. The utilization component alone (1.114) is only slightly above 1.0 and might not trigger scaling on its own. The TTFT component (1.700) is the dominant contributor, showing that response latency is the most pressing problem. Without the composite metric, a pure utilization-based policy might have decided that 78% is acceptable and held at 4 instances, leaving users experiencing slow responses.

Now using target tracking to compute how many instances are needed to bring utilization to 70%:

The system provisions one additional instance. Once it is ready (approximately 30-40 seconds for a 7B model), load balancing redistributes traffic across 5 instances. Average utilization should drop to approximately 62%, bringing the composite score below 1.0. If the 5-instance fleet still shows a composite score above 1.0 after the cooldown period, the policy will trigger again.

This example illustrates an important lesson: the target tracking formula based on utilization may suggest one additional instance, but the TTFT metric tells you that users are already experiencing degraded performance. A more sophisticated policy might use the TTFT exceedance as a signal to scale out by 2 instances rather than 1, accepting a brief cost increase in exchange for faster latency recovery.

Let us build a simulation of a reactive auto-scaling controller and visualize how different policies behave under a realistic LLM traffic pattern.

The cost-quality scatter reveals the fundamental tradeoff for every scaling policy. In this run, target tracking cuts dropped requests sharply without materially increasing average capacity relative to conservative threshold scaling. Step scaling eliminates drops entirely, but only by maintaining noticeably more capacity. The right operating point depends on the value of each request to your business: a service where each request represents a paid API call has very different economics than a service where requests are from free-tier users.

LLMs introduce scaling complications that do not appear in conventional web services. Understanding these challenges is necessary for building a scaling system that works in production. Many teams that have successfully auto-scaled conventional services discover that their existing tooling and intuitions need significant revision when applied to LLM workloads.

Transformer inference maintains a key-value cache for each active request. During the decode phase of a long-context conversation, the KV cache for a single request may contain hundreds of megabytes of state representing the model's attention computations over the entire conversation history. This state lives in GPU memory on the specific instance serving that request.

During auto-scaling decisions, it is tempting to simply terminate an instance when scaling in. But if that instance is currently serving long-context conversations, terminating it abruptly drops those requests and requires clients to retry from scratch. For a chatbot application where users are in the middle of multi-turn conversations, sudden termination means the conversation history is lost. The user experience is equivalent to a sudden crash.

There are two common mitigations. Graceful draining stops routing new requests to an instance that is marked for removal, then waits for all active requests to complete before terminating it. The instance continues serving its in-flight requests until they finish, then shuts down cleanly. For chat sessions, this can work well as long as session lengths are bounded. The risk is that very long-running sessions can prevent instances from draining for extended periods, tying up capacity that the policy has already decided to reclaim.

The second mitigation is KV cache migration: serializing the in-progress KV cache state and sending it to a remaining instance before terminating the source. This is technically complex: the KV cache must be serialized from GPU memory to CPU memory, transmitted over the network, and deserialized back into GPU memory on the target instance, all while the client connection is maintained. This requires specialized infrastructure and significantly increases the engineering complexity of the serving stack. However, it eliminates the request-loss problem entirely and allows scale-in without service disruption even for long-running conversations.

Prefix caching, discussed in Chapter 45.7, interacts with graceful draining in an important way. If the draining instance has cached the KV computations for a frequently used system prompt, allowing the instance to terminate means losing that cache. The receiving instances may experience a temporary performance degradation as they rebuild the cache from scratch.

As noted earlier, loading a large model into GPU memory is slow. On cloud platforms that provision bare-metal GPU instances on demand, there is also VM startup time on top of model loading. The combined cold start latency for a 70B parameter model can exceed three minutes in the worst case.

Practical production systems address this through pre-warmed instance pools: a small number of instances are kept running but unloaded (or loaded but idle), ready to start serving almost immediately when a scale-out event is triggered. The pool size is calibrated to absorb expected demand spikes without incurring too much idle cost. A typical approach is to maintain enough warm instances to handle the largest expected traffic ramp that reactive scaling alone cannot handle within the acceptable queue depth limit. If traffic can double in 2 minutes and instances take 3 minutes to start, you need a warm pool large enough to absorb that doubling without exceeding queue depth limits.

Container image caching is another critical optimization. If the model weights are baked into the container image and the image is cached on the host's local storage, loading time reduces to a local disk read rather than a network download from object storage. NVMe SSD storage on modern cloud instances provides read speeds of 5-10 GB/s, which is 5-10x faster than typical S3 or GCS download speeds. For a 14 GB 7B model in FP16, the difference is between roughly 1.5 seconds (local NVMe) and 14 seconds (network download).

Model weight snapshots extend this idea: instead of loading the model from scratch, you save a GPU memory snapshot after the model is loaded and warm, then restore from the snapshot for subsequent starts. NVIDIA's CUDA Driver API supports capturing and restoring GPU memory state, and some serving frameworks have built support for this pattern. The effective startup time then becomes the time to restore a memory snapshot, which can be under a second for models that fit in a single GPU.

Some production deployments use Kubernetes with custom admission controllers to schedule new model serving pods on nodes that already have the model weights cached on local disk. This turns model weight locality into a scheduling constraint, making sure that scale-out pods start on nodes where the startup will be fast.

LLM serving infrastructure is rarely homogeneous in large-scale deployments. You might have a base fleet of smaller GPU instances (A10G with 24 GB VRAM) supplemented by larger instances (A100 or H100 with 80 GB VRAM) for long-context requests that require more memory. A routing layer directs requests to the appropriate instance tier based on request characteristics.

Scaling policies need to be aware of instance types and must scale each tier independently. The scale-out trigger for the A10G tier (running out of VRAM due to long-context requests) is different from the trigger for the A100 tier (compute saturation due to high throughput). Applying a single fleet-wide utilization metric can lead to counterintuitive results: if A10Gs are 90% utilized and A100s are 30% utilized, a naive fleet-average of 60% might not trigger any scaling action, even though the A10G tier is under severe strain.

Scale-in decisions in heterogeneous fleets are also more complex. Terminating an A100 while keeping A10Gs running might not be correct if the remaining A10Gs cannot handle long-context requests. The scaling policy must account for both the overall fleet count and the minimum number of each instance type needed to serve all request types.

In Kubernetes, this is typically handled through node pools and node affinity rules. Each instance type is its own node pool with its own Horizontal Pod Autoscaler and its own scaling targets. Requests are routed to the appropriate pod pool based on a sidecar proxy or a custom admission webhook that inspects request headers.

For conventional web services, requests are roughly similar in cost: serving a 1 KB JSON response takes nearly the same compute as serving a 2 KB JSON response. For LLM services, request costs vary by orders of magnitude. A request to generate a 10-token response is perhaps 100x cheaper than a request to generate a 1,000-token response of the same model.

This variability means that request count is a poor unit of capacity planning. Two scenarios with identical request rates can have dramatically different compute requirements: 1,000 short requests per minute (average output: 50 tokens) and 1,000 long requests per minute (average output: 500 tokens) would require 10x different capacity. If your scaling metric is request rate or queue depth in terms of request count, you cannot distinguish these two scenarios.

The solution is token-based capacity planning. Instead of measuring requests per second, measure input tokens per second and output tokens per second separately. Each has a different cost profile: input tokens are processed in parallel during the prefill phase and are relatively cheap; output tokens are generated one at a time during the decode phase and are relatively expensive. A complete capacity model expresses the serving cost as a weighted sum of input token rate and output token rate, with the weights derived from profiling the model on your hardware.

Monitoring token rates also makes it possible to detect changes in request characteristics over time. If users start submitting much longer prompts without a corresponding increase in request count, the total token throughput will rise even though the request rate stays flat. A purely request-based monitoring system would miss this, while a token-based system would detect the increased load and trigger scaling before queues start growing.

When auto-scaling cannot respond fast enough to a sudden spike, traffic shaping becomes necessary. Rate limiting caps the number of requests accepted per time window per user or per API key. Request queuing with timeouts allows brief overload to be absorbed smoothly as instances come online. Priority queues let high-value requests jump ahead during constrained periods.

Rate limiting is typically implemented at the API gateway layer, before requests even reach the model serving infrastructure. When a user exceeds their rate limit, the API gateway returns a 429 Too Many Requests response with a Retry-After header indicating when the user can try again. This protects the serving fleet from individual users consuming disproportionate capacity during traffic spikes.

Request queuing with backpressure works differently: instead of rejecting requests that exceed current capacity, they are accepted into a queue and served in order as capacity becomes available. This is appropriate when clients can tolerate some delay and when the requests themselves are valuable enough to be worth queuing. The risk is queue depth growing unbounded during sustained overload. Queue depth limits and timeout policies ensure that requests that have waited too long are rejected rather than served stale.

Priority queues add a dimension of request importance. In a multi-tier pricing model, paid users might have higher priority than free-tier users. Time-sensitive requests (like real-time completions) might have higher priority than batch requests. During constrained periods, the priority queue ensures that high-value requests are served first, even if lower-priority requests must wait or be dropped.

Traffic shaping is not a substitute for good auto-scaling, but it is an essential complement. It prevents a short burst from causing cascading failures while the autoscaler catches up. The next chapter on caching explores how cached completions can further reduce the effective load that reaches the scaling layer.

Modern LLM serving frameworks like vLLM and TGI use continuous batching to maximize GPU utilization. In continuous batching, newly arrived requests are added to the current batch as soon as a slot opens up (when an existing request finishes generating a token), rather than waiting for an entire batch to complete before starting a new one. This dramatically improves throughput compared to static batching.

Continuous batching interacts with auto-scaling in subtle ways. When the serving engine is under-loaded, continuous batching may produce small batches (2-4 requests), which are GPU-inefficient but fast per request. When the serving engine is fully loaded, continuous batching produces large batches (16-32 requests), which are GPU-efficient but slower per request due to increased memory bandwidth contention.

This means that a single GPU utilization number does not fully capture the serving state. An instance running at 60% GPU utilization with a batch size of 4 is in a very different state from an instance running at 60% utilization with a batch size of 16. The first instance has room to absorb more requests without latency degradation; the second is already near its latency limit. Scaling policies that are aware of batch size can make better decisions, but batch size monitoring requires integration with the serving framework's internal metrics.

vLLM exports a rich set of metrics via its Prometheus endpoint, including current batch size, KV cache utilization, queue depth in tokens (not just requests), and per-request waiting times. Building auto-scaling policies on top of these framework-native metrics gives much more precise control than relying solely on hardware-level GPU utilization.

The key parameters for configuring auto-scaling policies encode your operational priorities and constraints:

• target_utilization: The desired GPU utilization fraction (typically 0.65-0.80). Lower values provide headroom for sudden spikes but increase idle cost. Services with strict latency SLAs often target 0.65-0.70; cost-optimized batch services may target 0.80-0.85.
• min_instances: The floor below which the fleet never scales. Set this high enough to handle minimum expected traffic without any warmup delay. For services that must respond instantly to traffic, set min_instances to the number needed to handle baseline load at the target utilization level.
• max_instances: The ceiling above which the fleet never scales. Prevents runaway cost during configuration errors or traffic anomalies. Should be set to the maximum your budget can sustain during a worst-case traffic event, not a typical peak.
• scale_out_cooldown: Minimum time between consecutive scale-out events. Should be at least as long as instance startup time to avoid repeatedly triggering scale-outs that the system is already handling.
• scale_in_cooldown: Minimum time between consecutive scale-in events. Usually 2-5x longer than scale-out cooldown to avoid removing capacity prematurely after a peak.
• startup_latency: The expected time from scale-out trigger to instance ready. Drives the minimum lead time for proactive scaling and the urgency of reactive scale-out.
• drain_timeout: Maximum time to wait for in-flight requests to complete before forcefully terminating a scale-in instance. For streaming completions, this should be at least the p99 completion time for your longest expected requests.
• warm_pool_size: The number of pre-warmed instances to maintain at all times for immediate scale-out. Higher values reduce cold-start latency during spikes at the cost of ongoing idle instance cost.

These parameters must be reviewed and updated regularly as traffic patterns change and model sizes evolve. A configuration that was appropriate for a 7B model may be entirely wrong after a migration to a 70B model with 4x longer startup times and 3x higher per-instance cost.

Auto-scaling solves the capacity mismatch problem but introduces its own failure modes. Understanding these limitations is as important as understanding the policies themselves, because the failure modes of a misconfigured autoscaler can be worse than running at fixed capacity.

The most dangerous failure mode is misconfigured scaling bounds. A max_instances cap that is set too low will allow the cluster to saturate under load even though the scaling policy is working correctly and demanding more instances. The policy will try to scale to 20 instances, hit the cap at 10, and the service will remain overloaded. From the outside, this looks like the autoscaler is failing to respond, but the root cause is a configuration error. A min_instances floor set too high wastes money continuously. Reviewing and updating these bounds is operational work that must happen regularly as traffic patterns and model sizes evolve.

The second significant limitation is metric reliability. If the monitoring system that reports GPU utilization or queue depth has an outage, the autoscaler loses its signal. A fault-tolerant implementation degrades gracefully in this scenario: either by defaulting to a conservative high instance count or by triggering a human alert rather than scaling down aggressively on a stale zero metric. Monitoring the monitoring system is not optional at scale. Some teams implement a "dead man's switch": if no valid metric data arrives for more than N seconds, the autoscaler defaults to a safe instance count rather than taking any scaling action.

Cold start latency is perhaps the most practically constraining limitation for LLM services specifically. Unlike a conventional web service where a new container starts in seconds, a large LLM instance can take minutes to become ready. This means auto-scaling cannot react instantaneously to sudden spikes, and some combination of over-provisioning, pre-warmed pools, and traffic shaping is always necessary. There is an inherent tension between cost efficiency and spike resilience that no amount of policy tuning can fully resolve. Services with tight latency SLAs during unpredictable load need to carry more warm capacity than services where some queue depth is acceptable.

The combination of high instance cost and long startup latency also creates a challenging failure recovery scenario. If a significant fraction of your serving fleet crashes simultaneously (due to a bad model update, a hardware fault, or a cascading software failure), restoring capacity takes much longer than in a conventional web service. A fleet of 20 A100 instances that all need to be restarted after a bad deployment will take minutes to recover, during which the service is severely degraded or completely down. This motivates a conservative deployment strategy: rolling updates that replace one instance at a time, with automatic rollback if the new version causes errors, rather than replacing the entire fleet simultaneously.

Auto-scaling in multi-tenant or multi-model deployments is substantially more complex than the single-model case. When dozens of models share a GPU cluster, scaling decisions for one model affect the resource availability for others. A sudden traffic spike to a popular model can starve less popular models of GPU capacity, causing their latency to degrade even though their own load has not changed. Scheduling systems like Kubernetes cluster autoscaler must be combined with per-model resource quotas and priority classes to prevent one popular model from crowding out all others.

Resource quotas in Kubernetes define the maximum CPU, memory, and GPU that any single model's pods can consume. Priority classes define which models get GPU resources first when the cluster is constrained. Combining these with namespace-scoped autoscalers allows each model to auto-scale within its quota without affecting other models. This is an active area of infrastructure engineering, and the tooling is maturing rapidly with the introduction of GPU-aware schedulers like NVIDIA's GPU Operator and the Kubernetes AI Framework.

Finally, cost attribution becomes important at scale. When GPU instances are shared across multiple models and scale independently, tracking the cost of serving each model requires careful instrumentation. Without cost attribution, it is impossible to know which models are expensive to serve and which scaling decisions are driving the infrastructure bill. A model that serves 1% of requests but consumes 20% of GPU capacity because it is disproportionately popular during expensive peak hours might be a candidate for rate limiting or tiered pricing, but you can only make that decision if you can see the cost breakdown.

Auto-scaling matches LLM serving capacity to demand automatically, reducing both over-provisioning cost and under-provisioning failures. The key takeaways from this chapter are:

• Metric selection determines everything: Choose metrics that directly reflect user-visible impact (TTFT, queue depth, token throughput) rather than infrastructure proxies alone (CPU load). Composite metrics reduce sensitivity to any single signal and provide a more complete picture of serving health.
• Scale-out is the primary strategy: Add more instances rather than larger instances. Horizontal scaling is incremental, resilient, and does not require service interruptions. Vertical scaling is a pre-deployment capacity planning decision, not a runtime scaling mechanism.
• Policies differ in responsiveness and stability: Threshold-based scaling is simple but prone to flapping. Target tracking is more stable, mathematically principled, and widely supported by cloud platforms. Step scaling handles sudden spikes aggressively by adding multiple instances at once when utilization is critically high.
• Proactive and reactive complement each other: Scheduled scaling handles predictable daily and weekly patterns; reactive policies handle deviations. Predictive scaling can further reduce cold-start gaps for traffic with learnable patterns, but must always be backstopped by reactive policies for anomalies.
• Cold start latency is the central engineering constraint for LLMs: Startup times measured in tens of seconds to minutes force a combination of pre-warmed pools, fast model loading via local caching and memory snapshots, and traffic shaping to handle spikes gracefully. Reducing startup latency directly improves the service's ability to respond to sudden demand changes.
• LLM-specific challenges require dedicated solutions: KV cache state makes instance termination more complex than in stateless services. Heterogeneous hardware fleets require per-tier scaling policies. Token-based capacity planning gives more accurate capacity estimates than request-based metrics. Continuous batching dynamics affect the relationship between utilization and latency in non-obvious ways.
• Scaling bounds and monitoring reliability are as important as the policy itself: A correctly implemented target tracking policy will fail silently if the metric it tracks is unreliable, or if the scaling bounds prevent it from reaching the desired instance count. Operational discipline around reviewing and updating configuration is a prerequisite for auto-scaling to work in practice.

In the next chapter on caching, we will see how returning cached completions for repeated prompts can dramatically reduce the effective request rate reaching the scaling layer, making auto-scaling cheaper and more stable.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about auto-scaling LLM production systems.