Auto-scaling › Key Parameters

The key parameters for configuring auto-scaling policies encode your operational priorities and constraints:

• target_utilization: The desired GPU utilization fraction (typically 0.65-0.80). Lower values provide headroom for sudden spikes but increase idle cost. Services with strict latency SLAs often target 0.65-0.70; cost-optimized batch services may target 0.80-0.85.
• min_instances: The floor below which the fleet never scales. Set this high enough to handle minimum expected traffic without any warmup delay. For services that must respond instantly to traffic, set min_instances to the number needed to handle baseline load at the target utilization level.
• max_instances: The ceiling above which the fleet never scales. Prevents runaway cost during configuration errors or traffic anomalies. Should be set to the maximum your budget can sustain during a worst-case traffic event, not a typical peak.
• scale_out_cooldown: Minimum time between consecutive scale-out events. Should be at least as long as instance startup time to avoid repeatedly triggering scale-outs that the system is already handling.
• scale_in_cooldown: Minimum time between consecutive scale-in events. Usually 2-5x longer than scale-out cooldown to avoid removing capacity prematurely after a peak.
• startup_latency: The expected time from scale-out trigger to instance ready. Drives the minimum lead time for proactive scaling and the urgency of reactive scale-out.
• drain_timeout: Maximum time to wait for in-flight requests to complete before forcefully terminating a scale-in instance. For streaming completions, this should be at least the p99 completion time for your longest expected requests.
• warm_pool_size: The number of pre-warmed instances to maintain at all times for immediate scale-out. Higher values reduce cold-start latency during spikes at the cost of ongoing idle instance cost.

These parameters must be reviewed and updated regularly as traffic patterns change and model sizes evolve. A configuration that was appropriate for a 7B model may be entirely wrong after a migration to a 70B model with 4x longer startup times and 3x higher per-instance cost.