Auto-scaling › Implementation
Language AI HandbookAll chapters Auto-scaling › ImplementationOne section · Sections in this chapter7 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us build a simulation of a reactive auto-scaling controller and visualize how different policies behave under a realistic LLM traffic pattern.
import numpy as np
# Simulate 24 hours of per-minute request arrivals
# Pattern: low overnight, ramp in morning, two midday peaks, evening decline
minutes = np.arange(0, 24 * 60)
hours = minutes / 60.0
base_rate = 20 # requests per minute at minimum
daily_pattern = (
base_rate
+ 80 * np.maximum(0, np.sin(np.pi * (hours - 7) / 12)) ** 2
+ 40 * np.exp(-0.5 * ((hours - 12.5) / 0.8) ** 2)
+ 25 * np.exp(-0.5 * ((hours - 15.0) / 1.0) ** 2)
)
# Add Poisson noise to simulate request arrival randomness
arrivals = np.random.poisson(daily_pattern)def simulate_autoscaler(
arrivals,
policy,
capacity_per_instance=40,
min_instances=2,
max_instances=16,
startup_minutes=3,
cooldown_minutes=5,
):
"""
Simulate auto-scaling under a given policy.
policy: callable(metric_value, target) -> desired_instance_delta
capacity_per_instance: max requests/min each instance can handle
"""
T = len(arrivals)
instances = np.zeros(T, dtype=int)
utilization = np.zeros(T)
queued = np.zeros(T)
served = np.zeros(T)
# State
current_instances = min_instances
pending = 0 # instances booting up
pending_ready_at = []
cooldown_remaining = 0
for t in range(T):
# Check if any pending instances are ready
newly_ready = sum(1 for r in pending_ready_at if r <= t)
current_instances += newly_ready
current_instances = min(current_instances, max_instances)
pending_ready_at = [r for r in pending_ready_at if r > t]
pending = len(pending_ready_at)
# Capacity and utilization
capacity = current_instances * capacity_per_instance
served_t = min(arrivals[t], capacity)
queue_t = max(0, arrivals[t] - capacity)
util_t = served_t / max(capacity, 1)
instances[t] = current_instances
utilization[t] = util_t
queued[t] = queue_t
served[t] = served_t
# Apply policy
if cooldown_remaining > 0:
cooldown_remaining -= 1
continue
desired = policy(
util_t, current_instances, min_instances, max_instances
)
delta = desired - current_instances
if delta > 0 and current_instances < max_instances:
# Scale out: schedule new instances
for _ in range(min(delta, max_instances - current_instances)):
pending_ready_at.append(t + startup_minutes)
cooldown_remaining = cooldown_minutes
elif delta < 0 and current_instances > min_instances:
current_instances = max(min_instances, current_instances + delta)
cooldown_remaining = cooldown_minutes
return instances, utilization, queued, served
# Define scaling policies
def threshold_policy(util, current, min_inst, max_inst, upper=0.99, lower=0.40):
if util > upper:
return current + 1
elif util < lower and current > min_inst:
return current - 1
return current
def target_tracking_policy(util, current, min_inst, max_inst, target=0.70):
desired = int(np.ceil(current * util / target))
return max(min_inst, min(max_inst, desired))
def step_policy(util, current, min_inst, max_inst):
if util > 0.85:
return min(max_inst, current + 4)
elif util > 0.75:
return min(max_inst, current + 2)
elif util > 0.65:
return min(max_inst, current + 1)
elif util < 0.30:
return max(min_inst, current - 1)
return current# Run simulations for all three policies
inst_thresh, util_thresh, queue_thresh, served_thresh = simulate_autoscaler(
arrivals, threshold_policy
)
inst_track, util_track, queue_track, served_track = simulate_autoscaler(
arrivals, target_tracking_policy
)
inst_step, util_step, queue_step, served_step = simulate_autoscaler(
arrivals, step_policy
)
hours_axis = minutes / 60.0
# Compute summary statistics
def dropped_requests(arrivals, served):
return arrivals - served
dropped_thresh = dropped_requests(arrivals, served_thresh).sum()
dropped_track = dropped_requests(arrivals, served_track).sum()
dropped_step = dropped_requests(arrivals, served_step).sum()
avg_inst_thresh = inst_thresh.mean()
avg_inst_track = inst_track.mean()
avg_inst_step = inst_step.mean()Policy Comparison Summary Policy Avg Instances Dropped Requests ------------------------------------------------------- Threshold 2.57 27 Target Tracking 2.57 5 Step Scaling 3.11 0
Each policy responds to the midday peak, but they differ in how aggressively they scale out. The conservative threshold policy waits until utilization is effectively saturated, keeping average capacity low but dropping the most requests during the ramp. Target tracking reacts earlier and drops fewer requests at similar average capacity. Step scaling is the most aggressive, provisioning extra headroom to eliminate almost all drops at a higher cost.
The startup latency chart illustrates why fast instance initialization matters so much for LLM auto-scaling. Under the same conservative threshold policy and three-stage synthetic ramp, each traffic step exposes the startup lag: the 1-minute case produces a brief, modest shortfall, the 2-minute case creates a taller and longer one, and the 5-minute case falls behind most severely. This motivates the engineering investment in pre-warmed instance pools and snapshot-based loading strategies, both of which are discussed in the next section.
The cost-quality scatter reveals the fundamental tradeoff for every scaling policy. In this run, target tracking cuts dropped requests sharply without materially increasing average capacity relative to conservative threshold scaling. Step scaling eliminates drops entirely, but only by maintaining noticeably more capacity. The right operating point depends on the value of each request to your business: a service where each request represents a paid API call has very different economics than a service where requests are from free-tier users.