Auto-scaling › Scaling Metrics

Before any scaling action can be triggered, the system needs a signal that reliably indicates whether current capacity is adequate. Choosing the right metric is the most consequential decision in auto-scaling design. The metric must be sensitive enough to detect overload early, stable enough not to trigger spurious scale events, and available with low enough latency to allow a timely response.

The challenge is that common infrastructure metrics often lag the actual user experience. A CPU utilization spike tells you the machine is busy, but it does not tell you whether requests are succeeding or how long they are taking. For LLM services, the relationship between infrastructure metrics and user-visible outcomes is especially indirect because inference workloads are highly variable: a three-token completion and a 4,096-token completion are radically different in resource consumption, yet both count as one request. A metric that works well for a homogeneous workload (like serving static files) can give completely misleading signals when applied to highly variable inference workloads.

Understanding the failure modes of each metric is as important as understanding what the metric measures. A metric that causes your autoscaler to oscillate is often worse than no autoscaler at all, because it adds the overhead of repeated scale events on top of whatever traffic problem it was meant to solve.

GPU utilization measures the fraction of time the GPU compute units are actively executing kernels. A utilization value near 1.0 generally means the GPU is saturated and additional requests will queue. A value near 0.0 means the GPU is idle and instances can be removed.

GPU utilization has an intuitive appeal because it measures the thing you are paying for: GPU compute time. If GPUs are running at full utilization, you need more GPUs. If they are nearly idle, you are paying for capacity you do not need. This logic holds well for compute-bound workloads, but LLM inference is often memory-bandwidth-bound rather than compute-bound. During the decoding phase of autoregressive generation, each forward pass reads model weights and KV cache entries from GPU memory and performs a relatively small number of floating-point operations on them. The ratio of memory reads to compute operations is high enough that GPU compute units may sit waiting for memory even while utilization metrics suggest the GPU is underloaded.

The limitation of GPU utilization as a scaling metric is memory pressure. An LLM inference server can exhaust its GPU memory (VRAM) while GPU compute utilization remains moderate, because the KV cache from many concurrent long-context requests fills memory before compute becomes the bottleneck. If a server is hosting 20 concurrent long-context conversations, each with a 32,000-token context window and a 4-billion parameter model using FP16 weights, the KV cache alone may consume tens of gigabytes of VRAM. If you scale on utilization alone, you will discover out-of-memory crashes before your autoscaler triggers a scale-out.

The practical workaround is to monitor GPU memory utilization separately from GPU compute utilization. VRAM utilization climbing above 85-90% is a signal that new requests will soon start failing with out-of-memory errors, even if compute utilization is only at 60%. Good production monitoring systems treat both as independent scaling triggers.

Queue depth counts how many requests are waiting for a free serving slot. Unlike utilization, it directly measures user-visible impact. A queue of zero means all requests are being served immediately. A growing queue means demand exceeds supply.

Queue depth is a lagging indicator by design: the queue can only grow after demand has already exceeded capacity. The practical implication is that scaling on queue depth alone requires generous look-ahead in the scaling policy. You need to add instances fast enough that the queue drains before it accumulates too many waiting requests. If your instances take 90 seconds to start, and you only trigger a scale-out when the queue exceeds 10 requests, the queue could grow to 50 or 100 requests before the new instance is ready. By then, many of those waiting requests may have timed out on the client side.

Queued requests and queued tokens measure different things. A request asking the model to summarize a 10,000-word document will occupy a serving slot for much longer than a request asking for a one-sentence answer. If your queue depth metric counts requests without regard to their size, the apparent queue depth may look the same for two situations that have vastly different workloads. More sophisticated serving frameworks like vLLM track queued tokens rather than queued requests, which gives a more accurate picture of the work backlog.

Queue depth is most useful when combined with other metrics. On its own, it can mask the difference between a single very large request blocking the queue and ten small requests all waiting for separate reasons.

Time to first token measures how long a user waits from submitting a request until the first token of the response appears. It is one of the most direct proxies for perceived latency, since users notice the delay before a streaming response begins. In a streaming API where partial responses are delivered token by token, users experience TTFT as the initial "pause" before the text starts flowing. Even if the overall generation speed is fast, a long TTFT makes the service feel slow and unresponsive.

TTFT has two components: queuing time and prefill time. Queuing time is the delay introduced by waiting for a free serving slot. Prefill time is the cost of processing the input tokens through the model before generation can start. For long prompts, prefill itself can take hundreds of milliseconds even on a fast GPU. For the purposes of scaling, the queuing component is more actionable: you can reduce it by adding capacity. The prefill component is fixed by the model and hardware.

TTFT increases when the serving cluster is overloaded because requests spend more time waiting in queue before prefill begins. It is therefore a useful scaling metric: if p95 TTFT rises above a target threshold, the system needs more capacity. P95 (the 95th percentile) is typically more appropriate than the average or median here, because the worst 5% of users have the worst experience, and if that experience degrades significantly, you will hear about it in user complaints before it shows up in average metrics.

A practical complication is that TTFT varies significantly with prompt length. If your users submit prompts of wildly different lengths, the p95 TTFT will fluctuate for reasons unrelated to capacity. Services that want to use TTFT as a scaling metric often normalize it by prompt length, computing something like TTFT per 1,000 input tokens.

This metric normalizes throughput by the number of running instances. As we covered in Chapter 45.4, throughput optimization is central to LLM serving economics. Monitoring tokens-per-second-per-instance lets you detect when individual instances are underperforming (perhaps due to memory pressure) and when the fleet as a whole is approaching saturation.

When tokens-per-second-per-instance is declining while instance count is constant, it indicates that requests are becoming harder (longer context, more concurrent requests per instance) or that hardware is degrading (thermal throttling, ECC errors slowing memory access). This metric can therefore diagnose issues that pure utilization metrics miss. If utilization is flat but throughput per instance is declining, you have a quality problem rather than a capacity problem.

At the fleet level, tracking total tokens-per-second against an expected value derived from your peak-hour forecasts gives you a quick sanity check that the fleet is keeping up with demand. This is particularly useful for batch inference workloads where you have a known token budget to process within a deadline.

Inter-token latency, also called time between tokens or generation speed, measures how long the model takes to produce each successive token after the first. While TTFT captures the initial wait, ITL governs the streaming experience: a fast ITL makes responses feel fluid and fast, while a high ITL causes the streaming text to appear in slow, choppy bursts.

ITL is determined primarily by GPU compute and memory bandwidth, not by queue depth. It changes relatively little as a function of load, because each decoding step involves a forward pass through the model that takes roughly the same time regardless of how many requests are in the queue. The exception is continuous batching: when many requests are batched together during the decode phase, each individual request's ITL degrades somewhat because the batch is larger and the memory bandwidth is shared.

Including ITL in your scaling metrics helps detect when the serving system is over-batching: packing too many requests into each batch in an attempt to maximize throughput at the cost of per-request latency. A rising ITL alongside a stable TTFT can indicate this condition.

In practice, the most robust scaling signal is a composite of several indicators. A common approach is to define a scaling score that weighs GPU utilization, queue depth, and p95 TTFT simultaneously:

$$\text{scaling\_score} = w_1 \cdot \frac{\text{util}}{\text{util\_target}} + w_2 \cdot \frac{\text{queue}}{\text{queue\_target}} + w_3 \cdot \frac{\text{TTFT}}{\text{TTFT\_target}}$$

where:

• $\text{util}$ is the current GPU utilization averaged across the fleet
• $\text{util\_target}$ is the target utilization you want to operate at (often 0.75)
• $\text{queue}$ is the current request queue depth
• $\text{queue\_target}$ is the maximum acceptable queue depth
• $\text{TTFT}$ is the current p95 time to first token in milliseconds
• $\text{TTFT\_target}$ is the SLA target for time to first token
• $w_1, w_2, w_3$ are weights that sum to 1.0 and reflect how much each metric matters for your service

When the composite score exceeds 1.0, the system scales out. When it drops below a low threshold (often 0.5), the system scales in. The composite approach prevents any single metric from causing oscillation while remaining sensitive to the combination of signals that together indicate strain.

The weights $w_1, w_2, w_3$ encode your priorities. A service with a strict latency SLA might assign a high weight to TTFT: $w_3 = 0.6$, leaving the other two metrics as secondary confirmations. A batch processing service that cares more about throughput than latency might weight utilization most heavily: $w_1 = 0.6$. There is no universally correct weighting; it must be calibrated against your specific service requirements.

One important implementation detail: the metrics in the composite score should be smoothed over a rolling window before comparison. Raw per-second GPU utilization is noisy. A single second of 95% utilization during a burst of long prefills might not warrant a scale-out. A rolling average over 60-90 seconds gives a more stable signal. The window length is a tradeoff between responsiveness (shorter window) and stability (longer window).