Auto-scaling › Scaling Policies in Practice: A Worked Example

To make these concepts concrete, let us trace through a complete scaling decision for a hypothetical LLM service. The service runs a 7B parameter model on A10G instances (24 GB VRAM each). Current state: 4 instances running, 78% average GPU utilization, a request queue depth of 3, and p95 TTFT of 850 ms against a target of 500 ms.

The composite scaling score combines all three metrics. The weights here reflect a service that prioritizes latency above utilization: $w_1 = 0.4$ for utilization, $w_2 = 0.3$ for queue depth, and $w_3 = 0.3$ for TTFT, with targets of 0.70, 5 queued requests, and 500 ms respectively.

$$\text{score} = 0.4 \cdot \frac{0.78}{0.70} + 0.3 \cdot \frac{3}{5} + 0.3 \cdot \frac{850}{500}$$ $$= 0.4 \cdot 1.114 + 0.3 \cdot 0.600 + 0.3 \cdot 1.700$$ $$= 0.446 + 0.180 + 0.510 = 1.136$$

The score exceeds 1.0, so a scale-out is triggered. The utilization component alone (1.114) is only slightly above 1.0 and might not trigger scaling on its own. The TTFT component (1.700) is the dominant contributor, showing that response latency is the most pressing problem. Without the composite metric, a pure utilization-based policy might have decided that 78% is acceptable and held at 4 instances, leaving users experiencing slow responses.

Now using target tracking to compute how many instances are needed to bring utilization to 70%:

$$\text{desired} = \left\lceil 4 \times \frac{0.78}{0.70} \right\rceil = \lceil 4.457 \rceil = 5$$

The system provisions one additional instance. Once it is ready (approximately 30-40 seconds for a 7B model), load balancing redistributes traffic across 5 instances. Average utilization should drop to approximately 62%, bringing the composite score below 1.0. If the 5-instance fleet still shows a composite score above 1.0 after the cooldown period, the policy will trigger again.

This example illustrates an important lesson: the target tracking formula based on utilization may suggest one additional instance, but the TTFT metric tells you that users are already experiencing degraded performance. A more sophisticated policy might use the TTFT exceedance as a signal to scale out by 2 instances rather than 1, accepting a brief cost increase in exchange for faster latency recovery.