LLM Inference Serving: Architecture, Routing & Auto-Scaling › Auto-Scaling
Language AI HandbookAll chapters LLM Inference Serving: Architecture, Routing & Auto-Scaling › Auto-ScalingOne section · Sections in this chapter5 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Static deployments with a fixed number of instances struggle with variable demand. Traffic to inference services often varies dramatically by time of day, day of week, and in response to external events. Auto-scaling adjusts the number of model instances based on load, reducing costs during low-traffic periods while maintaining performance during peaks. This elastic capacity is one of the key advantages of cloud deployments over fixed on-premises infrastructure.
Think of auto-scaling as hiring and laying off staff in real time. A restaurant that always staffs for the dinner rush pays full labor costs all day even when it's empty at 3pm. One that can bring servers in two hours before the rush and send them home after closing minimizes labor cost without sacrificing service quality during peak demand. Auto-scaling does the same for GPU compute: you pay for instances only when you need them, rather than always provisioning for worst-case load. The challenge is that unlike restaurant staff who can show up in minutes, LLM instances take substantial time to become ready, which makes anticipating demand spikes necessary.
The decision logic for auto-scaling involves two separate problems: detecting when to scale (policy) and how fast to scale (mechanism). The policy problem is about choosing the right metrics and thresholds that accurately predict when current capacity will be insufficient. Scaling too eagerly wastes money; scaling too timidly allows user-facing degradation. The mechanism problem is about how quickly new instances become available and how smoothly traffic shifts to them. Cold starts that take two minutes create a significant gap between when you decide to scale and when the capacity is available.
A subtle challenge in auto-scaling LLM services is that the feedback loop between scaling decisions and their effects is delayed. When you add a new instance, it takes time to load the model, warm its caches, and receive traffic. During this time, the original instances continue experiencing high load. This delay means scaling logic must be predictive rather than purely reactive: you need to decide to scale before the situation becomes necessary, not after users are already experiencing degradation.
Scaling Metrics
Choosing the right metrics to trigger scaling is important because different metrics capture different aspects of system health and have different response characteristics.
Queue depth is the number of waiting requests and provides a direct measure of demand exceeding capacity. High queue depth indicates insufficient capacity and directly predicts increased latency for arriving requests. This metric responds quickly to load changes because queues grow immediately when arrivals exceed processing rate. The key insight is that a deep queue is a leading indicator of latency problems: if you wait until P95 latency is already high, requests have already waited too long. Acting on queue depth lets you scale preemptively before users experience degradation.
Latency percentiles, specifically P50, P95, or P99 request latency, directly measure user experience. Scaling based on latency targets addresses user experience directly, which is ultimately what matters. However, latency increases only after the system is already overloaded, making this a lagging indicator. By the time latency rises, users have already experienced degraded service, so latency metrics are better for detecting that scaling has failed than for triggering timely scale-up actions.
GPU utilization is the fraction of GPU compute being used and indicates how efficiently the current capacity is being used. Low utilization suggests over-provisioning and wasted cost. Consistently high utilization suggests under-provisioning and risk of latency degradation as load increases further. However, GPU utilization can be misleading for LLM serving: a server spending most of its time on memory-bound decode operations may show low compute utilization while still being fully saturated in terms of memory bandwidth.
Tokens per second is the aggregate throughput across instances and measures actual work being done. Unlike request count, this metric accounts for variable request sizes. A spike in tokens per second indicates increased load even if request count is stable (perhaps because users are submitting longer prompts).
from dataclasses import dataclass
@dataclass
class ScalingMetrics:
"""Metrics used for auto-scaling decisions."""
queue_depth: int
active_requests: int
latency_p50_ms: float
latency_p95_ms: float
gpu_utilization: float
tokens_per_second: float
@dataclass
class ScalingConfig:
"""Thresholds for scaling decisions."""
# Scale up triggers
max_queue_depth: int = 100
max_latency_p95_ms: float = 5000
max_gpu_utilization: float = 0.85
# Scale down triggers
min_queue_depth: int = 10
min_gpu_utilization: float = 0.30
# Cooldown to prevent thrashing
scale_up_cooldown_seconds: float = 60
scale_down_cooldown_seconds: float = 300
# Limits
min_replicas: int = 1
max_replicas: int = 10
class AutoScaler:
"""Determines scaling actions based on metrics."""
def __init__(self, config: ScalingConfig):
self.config = config
self.last_scale_up = 0.0
self.last_scale_down = 0.0
self.current_replicas = 1
def evaluate(self, metrics: ScalingMetrics, current_time: float) -> int:
"""
Returns desired replica count.
Positive delta = scale up, negative = scale down.
"""
# Check scale up conditions
should_scale_up = (
metrics.queue_depth > self.config.max_queue_depth
or metrics.latency_p95_ms > self.config.max_latency_p95_ms
or metrics.gpu_utilization > self.config.max_gpu_utilization
)
# Check scale down conditions
should_scale_down = (
metrics.queue_depth < self.config.min_queue_depth
and metrics.gpu_utilization < self.config.min_gpu_utilization
)
# Apply cooldowns
time_since_up = current_time - self.last_scale_up
time_since_down = current_time - self.last_scale_down
if (
should_scale_up
and time_since_up > self.config.scale_up_cooldown_seconds
):
new_replicas = min(
self.current_replicas + 1, self.config.max_replicas
)
if new_replicas > self.current_replicas:
self.last_scale_up = current_time
self.current_replicas = new_replicas
elif (
should_scale_down
and time_since_down > self.config.scale_down_cooldown_seconds
):
new_replicas = max(
self.current_replicas - 1, self.config.min_replicas
)
if new_replicas < self.current_replicas:
self.last_scale_down = current_time
self.current_replicas = new_replicas
return self.current_replicas# Simulate auto-scaling over time
scaling_config = ScalingConfig(
max_queue_depth=50, max_latency_p95_ms=3000, min_replicas=1, max_replicas=5
)
scaler = AutoScaler(scaling_config)
# Simulate increasing load
scenarios = [
(0, ScalingMetrics(10, 5, 500, 1000, 0.3, 100)), # Low load
(120, ScalingMetrics(60, 30, 1000, 2500, 0.7, 200)), # Growing
(240, ScalingMetrics(100, 50, 2000, 4000, 0.9, 250)), # Overloaded
(360, ScalingMetrics(80, 40, 1500, 3500, 0.85, 400)), # Still high
(480, ScalingMetrics(75, 35, 1400, 3200, 0.88, 380)), # Peak sustained
(600, ScalingMetrics(20, 10, 500, 1000, 0.4, 300)), # Recovered
(1000, ScalingMetrics(5, 2, 300, 500, 0.2, 100)), # Low load
]
scaling_results = []
for time_sec, metrics in scenarios:
replicas = scaler.evaluate(metrics, time_sec)
scaling_results.append((time_sec, metrics, replicas))Auto-scaling Simulation: ------------------------------------------------------------ t= 0s | queue= 10 | p95=1000ms | gpu=30% | replicas=1 t= 120s | queue= 60 | p95=2500ms | gpu=70% | replicas=2 t= 240s | queue=100 | p95=4000ms | gpu=90% | replicas=3 t= 360s | queue= 80 | p95=3500ms | gpu=85% | replicas=4 t= 480s | queue= 75 | p95=3200ms | gpu=88% | replicas=5 t= 600s | queue= 20 | p95=1000ms | gpu=40% | replicas=5 t=1000s | queue= 5 | p95=500ms | gpu=20% | replicas=4
The simulation illustrates key auto-scaling behaviors. The system scales up quickly when queue depth spikes, adding capacity before latency degrades too severely. It then gradually scales down as load decreases, but with a longer cooldown to avoid oscillation. The asymmetric cooldowns (60 seconds up, 300 seconds down) prevent thrashing while so fast response to traffic increases. Scaling up quickly prevents user-facing degradation, while scaling down slowly ensures that brief dips in traffic don't cause premature capacity reduction that would require expensive cold starts when traffic rebounds.
The key parameters for the auto-scaling logic are:
- max_queue_depth: Threshold for the number of queued requests that triggers a scale-up event. Setting this too low causes unnecessary scaling; setting it too high delays scaling until users are already experiencing degradation.
- max_latency_p95_ms: P95 latency threshold that triggers scaling when exceeded. Because latency is a lagging indicator, this typically fires after queue depth has already signaled the problem.
- scale_up_cooldown_seconds: Minimum time interval between scale-up actions to prevent flapping. Should be long enough that new instances become ready before the next scaling evaluation, but short enough to allow rapid scale-up when needed.
Cold Start Challenges
LLM instances take significant time to become ready for serving. Loading a 7B parameter model requires transferring approximately 14GB (in FP16) from storage to GPU memory, then initializing CUDA kernels and compiling any JIT-compiled operations, and allocating KV cache space. This cold start latency, often 30-120 seconds for typical deployments, complicates auto-scaling because new capacity isn't immediately available when scaling decisions are made.
Think of cold start latency as the time it takes a new employee to get their computer set up and learn the login credentials before they can start taking calls. You can hire them the moment the phone lines get busy, but they won't be answering calls for the first hour. For LLM serving, this means you need to start scaling before demand spikes, not in response to demand spikes. Reactive scaling that triggers only when queues are already long will consistently fail to provide capacity in time.
The 30-120 second cold start range spans multiple orders of magnitude depending on deployment configuration. A 7B model stored on local NVMe drives loads much faster than the same model stored in network-attached storage. Models that use eager CUDA kernel compilation (like some quantized models) add seconds of initialization overhead. Models with large context windows allocate more KV cache memory, which takes longer to initialize. Understanding the specific cold start profile of your deployment is needed for configuring appropriate scale-up lead times.
Several strategies help mitigate cold start impact.
Warm pools maintain a pool of pre-initialized but idle instances that have already loaded model weights and initialized their CUDA context. New traffic immediately activates these instances rather than waiting for cold start. The trade-off is paying for idle GPU time, as warm pool instances consume resources even when not serving requests.
Predictive scaling uses historical patterns to anticipate demand and scale preemptively. If traffic typically peaks at 9 AM on weekdays, begin scaling at 8:50 AM so new instances are ready before the rush. Machine learning models can predict traffic patterns more accurately than simple rules, but require historical data and ongoing maintenance.
Gradual traffic shift brings new instances online by ramping traffic gradually rather than immediately directing full load. This prevents overwhelming a fresh instance while its various caches (file system cache, KV prefix cache, CUDA kernel cache) are cold. Gradual ramp-up also allows detecting problems with new instances before they impact a large fraction of traffic.
import time
from typing import Optional
@dataclass
class WarmPoolInstance:
"""Instance in warm pool, ready for activation."""
instance_id: str
created_time: float
state: str = "warm" # warm, activating, active, draining
class WarmPoolManager:
"""Manages a pool of pre-warmed instances."""
def __init__(
self,
min_warm: int = 2,
max_warm: int = 5,
warm_instance_ttl: float = 3600, # 1 hour
):
self.min_warm = min_warm
self.max_warm = max_warm
self.warm_instance_ttl = warm_instance_ttl
self.warm_pool: List[WarmPoolInstance] = []
self.active_instances: List[WarmPoolInstance] = []
def activate_instance(self) -> Optional[WarmPoolInstance]:
"""
Activate a warm instance for serving.
Returns None if pool is empty.
"""
if not self.warm_pool:
return None
instance = self.warm_pool.pop(0)
instance.state = "activating"
self.active_instances.append(instance)
return instance
def deactivate_instance(self, instance_id: str):
"""Return instance to warm pool instead of terminating."""
for i, inst in enumerate(self.active_instances):
if inst.instance_id == instance_id:
inst.state = "warm"
inst.created_time = time.time() # Reset TTL
self.active_instances.pop(i)
if len(self.warm_pool) < self.max_warm:
self.warm_pool.append(inst)
return
def pool_status(self) -> dict:
return {
"warm_count": len(self.warm_pool),
"active_count": len(self.active_instances),
"warm_ids": [i.instance_id for i in self.warm_pool],
}# Demonstrate warm pool behavior
pool = WarmPoolManager(min_warm=2, max_warm=4)
# Pre-populate warm pool
for i in range(3):
pool.warm_pool.append(
WarmPoolInstance(instance_id=f"inst-{i}", created_time=time.time())
)
initial_status = pool.pool_status()
# Traffic spike: activate instances quickly
activated_instances = []
for i in range(2):
inst = pool.activate_instance()
if inst:
activated_instances.append(inst.instance_id)
spike_status = pool.pool_status()
# Traffic subsides: return to warm pool
pool.deactivate_instance("inst-0")
subside_status = pool.pool_status()Initial State:
{'warm_count': 3, 'active_count': 0, 'warm_ids': ['inst-0', 'inst-1', 'inst-2']}
Traffic spike - activating instances:
Activated: inst-0
Activated: inst-1
Pool: {'warm_count': 1, 'active_count': 2, 'warm_ids': ['inst-2']}
Traffic subsides - returning instances:
Deactivated: inst-0
Pool: {'warm_count': 2, 'active_count': 1, 'warm_ids': ['inst-2', 'inst-0']}The warm pool enables near-instant scaling for traffic spikes. When traffic increases, instances activate in seconds (just the time to update routing tables) rather than the minute or more required for cold start. When traffic drops, instances return to the warm pool rather than terminating completely, preserving their warmed state for the next spike. This approach trades some ongoing cost (idle warm instances) for dramatically improved responsiveness to load changes.