Auto-scaling › Proactive Scaling Policies

Reactive policies always trail demand by at least one metric collection cycle plus instance startup time. For LLMs with their substantial startup latency, even a 60-second delay between the metric crossing a threshold and a new instance becoming ready can mean hundreds of dropped requests. Proactive scaling eliminates the lag for predictable load patterns by adding capacity before demand arrives.

Proactive and reactive policies are complements, not alternatives. Proactive scaling handles the predictable patterns that repeat daily and weekly. Reactive scaling handles the deviations from those patterns: an unexpectedly busy Monday, a viral mention that drives a traffic spike during an otherwise quiet period, or a scheduled maintenance window that compresses traffic into a smaller time window.

Scheduled scaling sets a minimum instance count at specific times. If your LLM service handles enterprise customers who all log in between 9 AM and 10 AM in their local timezone, you can schedule a scale-out at 8:45 AM and a scale-in at 7 PM.

Scheduled scaling is reliable for predictable traffic patterns and does not require any metric measurement. Its weakness is that it does not adapt to anomalies. If a scheduled event is cancelled and traffic is unexpectedly low, the scaled-out fleet wastes money. If an unscheduled event creates traffic outside of the scheduled window, the schedule provides no help.

In practice, scheduled scaling is used as a floor for reactive scaling: you schedule a minimum capacity that handles expected load, and reactive policies handle deviations above and below it. The scheduled minimum prevents the reactive policy from scaling in too aggressively during a brief lull between expected traffic peaks.

Timezone management is a practical concern that becomes important at scale. An enterprise service with customers in multiple continents may have overlapping business-hours windows that create a rolling "wave" of scheduled peaks across a 24-hour period. Each timezone transition needs to be accounted for in the schedule, or the schedule quickly becomes incorrect during daylight saving time transitions.

Predictive scaling trains a forecasting model on historical traffic data to predict future load, then provisions instances in advance based on those predictions. The forecast horizon must be at least as long as the instance startup time to be useful. For a system where instances take 3 minutes to be ready, you need a forecast that is accurate 3 minutes into the future. For practical safety margin, you want a forecast horizon of 5-10 minutes.

For daily and weekly patterns, even simple forecasting models (ARIMA, exponential smoothing, or a basic neural network) can achieve enough accuracy to substantially reduce cold-start gaps. The key insight is that you do not need to predict the exact traffic level, only whether you need more capacity than you currently have. A conservative forecast that over-predicts load by 20% is often preferable to a precise forecast that is occasionally wrong in the under-prediction direction.

Cloud providers like AWS offer predictive scaling as a managed feature that analyzes historical CloudWatch metrics and generates hourly forecasts automatically. Google Cloud also offers predictive autoscaling based on historical load patterns. These managed services reduce the engineering burden of building and maintaining a custom forecasting pipeline.

The critical failure mode of predictive scaling is distribution shift: if a product launch, a marketing campaign, or a news event generates traffic unlike any historical pattern, the predictor will under-forecast. A model trained on two months of historical data has never seen the traffic pattern created by being mentioned on a major news site. Predictive scaling should always be combined with reactive policies as a safety net for unforeseen spikes. The combination is: predictive scaling ensures capacity is ready for the expected pattern, reactive scaling handles everything outside that pattern.