Auto-scaling › Reactive Scaling Policies

A scaling policy defines the rules that translate metric observations into scaling actions. Reactive policies respond to what the system is currently experiencing. They are the foundation of any auto-scaling system and handle the majority of scaling events in most production deployments.

The core challenge in designing reactive policies is balancing responsiveness against stability. A policy that reacts instantly to every metric fluctuation will produce a system that constantly adds and removes instances, wasting the startup time of each instance it provisions and never reaching a steady state. A policy that waits too long to react will allow queue depth and latency to grow for extended periods before relief arrives. Good reactive policies are tuned to respond quickly to meaningful load changes while ignoring transient noise.

The simplest reactive policy scales out when a metric crosses an upper threshold and scales in when it crosses a lower threshold. For example: add one instance when GPU utilization exceeds 80%, remove one instance when it falls below 30%.

Threshold-based scaling is easy to understand and implement. Its weakness is oscillation: if the metric hovers near the threshold, the system may repeatedly add and remove instances, wasting the startup time of each instance it provisions. This is called "flapping." Flapping incurs constant startup costs without allowing the system to stabilize, and the repeated scale-in and scale-out events can cause service disruptions as the load balancer redistributes traffic each time.

Cooldown periods are the standard defense against flapping. A cooldown period is a mandatory waiting time after any scaling action before the next one is allowed. If scale-out takes 60 seconds to provision and stabilize a new instance, a 120-second cooldown prevents triggering another scale-out before the first one has had a chance to reduce the metric.

Separate cooldowns for scale-out and scale-in are an important refinement. Scale-out cooldowns are typically shorter because you want to respond quickly to load growth. Scale-in cooldowns are typically longer because premature scale-in wastes the startup cost of the instance that was just added. A common configuration might use a 90-second scale-out cooldown and a 300-second scale-in cooldown.

The hysteresis gap (the difference between the scale-out threshold and the scale-in threshold) also prevents flapping. In the example above, the 50-percentage-point gap between the 80% scale-out threshold and the 30% scale-in threshold means that a new instance, once added, must reduce utilization below the lower threshold before the system considers removing it again.

Step scaling generalizes threshold-based policies by defining multiple thresholds, each triggering a different number of instances. For example:

• Utilization between 80% and 90%: add 1 instance
• Utilization between 90% and 95%: add 2 instances
• Utilization above 95%: add 4 instances

This allows the system to respond more aggressively to severe overload while avoiding over-reaction to mild load increases. Step scaling is well-suited to traffic patterns with sudden, large spikes: if traffic doubles in 60 seconds, a step policy can add 4 instances at once instead of slowly adding them one at a time through repeated threshold crossings.

The step boundaries require tuning for each service. A poorly calibrated step policy with large steps can over-provision massively during a transient spike, wasting significant money before the scale-in cooldown allows the fleet to shrink. Step policies work best when you have well-understood traffic distributions and have tested the policy under realistic load scenarios.

For LLMs specifically, step scaling during the scale-out direction is often appropriate because the cost of adding instances is high (both in actual GPU cost and in the coordination overhead of redistributing load) and you want to avoid repeatedly adding one instance at a time during a rapid traffic ramp. Adding 4 instances at once when utilization hits 95% is often more efficient than the escalating sequence of threshold crossings that a simpler policy would generate.

Target tracking inverts the logic: instead of defining thresholds and actions, you define a target value for the metric and let the scaler continuously adjust instance count to maintain it. If the target is 70% GPU utilization and current utilization is 90%, the scaler computes how many additional instances are needed to bring average utilization back to 70%.

$$\text{desired\_instances} = \left\lceil \text{current\_instances} \times \frac{\text{current\_metric}}{\text{target\_metric}} \right\rceil$$

where:

• $\text{desired\_instances}$ is the computed number of instances needed
• $\text{current\_instances}$ is how many instances are running now
• $\text{current\_metric}$ is the current metric value (e.g., utilization = 0.90)
• $\text{target\_metric}$ is the desired metric value (e.g., 0.70)
• $\lceil \cdot \rceil$ is the ceiling function, since you cannot run fractional instances

This formula gives a proportional response: the further the current metric is from the target, the larger the adjustment. At moderate overload (say, 84% vs 70% target with 5 running instances), the formula suggests 6 instances. At severe overload (95% vs 70% target with 5 instances), it suggests 7 instances. The response scales continuously with the severity of the overload.

Target tracking is the most widely used policy in production because it naturally handles a wide range of load levels without requiring manual threshold tuning. AWS Auto Scaling Groups, Kubernetes Horizontal Pod Autoscaler, and Google Cloud Managed Instance Groups all offer target tracking as a first-class policy. You specify the target metric value and let the autoscaler do the arithmetic.

One subtlety of target tracking for LLMs: the formula above assumes a linear relationship between instance count and metric value. If adding one instance reduces average utilization from 90% to 78%, target tracking will work well. But LLM workloads sometimes exhibit non-linear relationships due to continuous batching. Adding more serving capacity can allow the serving engine to batch more requests together, which can reduce per-request throughput (because each request waits longer for a full batch) while increasing overall tokens-per-second. In these situations, target tracking on utilization alone may not converge to the expected operating point, and the composite metric approach becomes more important.