Throughput Optimization › Measuring Throughput

Before you optimize throughput, you need to measure it correctly. This sounds simple but involves several subtleties that can lead to misleading benchmarks if you are not careful. A benchmark that overstates throughput by 3x will cause you to under-provision hardware and produce an outage the first time real users arrive.

Language model throughput can be measured at two granularities:

• Request throughput: the number of complete request-response cycles per second. This is the most relevant metric for capacity planning, since each request represents a user interaction.
• Token throughput: the total number of tokens generated per second across all concurrent requests. This is more useful for understanding hardware utilization, since the GPU generates tokens regardless of how they group into requests.

The two metrics are related. If your average response contains $L$ tokens and your system processes $R$ requests per second, then token throughput is approximately $R \times L$. However, token throughput depends on output length, which varies by request. A system serving short answers shows different token throughput than one generating long documents, even at identical GPU utilization.

For capacity planning, always report both. Token throughput tells you how close you are to the hardware ceiling. Request throughput tells you how many users you can serve. A system running at 10,000 tokens/second with an average output of 100 tokens handles 100 requests per second. The same system with an average output of 500 tokens handles only 20 requests per second, even though GPU utilization is identical.

Time-to-first-token (TTFT) and inter-token latency (ITL) are also worth tracking alongside throughput. TTFT measures how quickly the model starts generating after receiving a request, which depends on prefill compute time and queueing delays. ITL measures the gap between successive generated tokens, which reflects the decode step time. Users perceive both: a long TTFT feels like the system is hanging even if generation is fast once it starts, while high ITL makes streaming responses feel choppy.

A subtle but important distinction: the throughput you measure in isolation differs from the throughput you achieve under sustained load.

Peak throughput is the maximum rate when the system is fully saturated: every GPU thread is busy, batches are full, and no resources are idle. Benchmarks that ramp up load until the system saturates measure this.

Sustained throughput is the rate you can maintain over minutes or hours without degradation. It is limited by memory accumulation (KV cache growth), garbage collection overhead in the serving framework, and thermal throttling on some hardware. On many systems, sustained throughput runs 10-20% below peak because of these ongoing overheads.

Burst throughput is the rate achievable for short windows when internal queues are empty and memory is fresh. It may exceed sustained throughput because the system has not yet accumulated overhead. Burst throughput is what many public benchmarks report, making them optimistic for real production planning.

Production planning should use sustained throughput, measured over at least five minutes of continuous load, not brief benchmark windows. The difference between burst and sustained throughput is the difference between a benchmark and reality.

Throughput and latency are not independent: they form a curve. At low request rates, the system is underloaded. Latency is near minimum, dominated by a single request's computation, but throughput is low because the GPU is idle much of the time. The CPU is submitting work faster than it accumulates, so the GPU sits waiting.

As request rate increases, the system fills its batch capacity. Latency increases because requests queue for batch slots, but throughput rises because the GPU is now busier. Eventually the system saturates: latency climbs steeply while throughput plateaus. Beyond saturation, adding more requests only causes queuing without improving throughput. The queue grows without bound and latency approaches infinity in the mathematical model, or your queue drops requests in the practical implementation.

The optimal operating point for most production systems is just below saturation: throughput is near-maximum, and latency is still acceptable. This is not a fixed point on the curve: it moves based on your latency SLO. A system with a 2-second SLO can run at higher utilization than one with a 100-millisecond SLO, because the latency budget absorbs more queuing delay.

Understanding the shape of this curve for your specific model and hardware is essential before setting capacity targets. Systems with highly variable output lengths (like chat models) have broader curves because continuous batching smooths the saturation behavior. Systems with fixed output lengths (like code completion at a fixed number of tokens) have sharper saturation transitions.

The curve in @fig-throughput-latency-curve illustrates the fundamental tension. Notice how throughput rises nearly linearly until saturation and then plateaus, while latency stays low until the system is about 80% loaded and then climbs steeply. The green shading marks the practical operating zone: you want to keep the system loaded enough to maximize GPU utilization but not so loaded that latency blows up.

What the curve does not show is how it shifts under different conditions. Adding GPUs shifts the saturation point rightward without changing the curve shape. Enabling continuous batching shifts the saturation point rightward and also changes the steepness of the latency rise, because batch slots refill continuously rather than discretely. Understanding both effects helps you reason about the return on investment of each optimization.

A common mistake in throughput benchmarking is reporting average latency alongside throughput. For production capacity planning, percentile latencies matter far more. If your system serves 1,000 requests per second with a mean latency of 200 ms but a P99 latency of 4 seconds, then 10 users per second experience a 4-second wait. In a production API, P99 latency frequently triggers SLO violations and user complaints.

The reason percentile latencies diverge from averages under load is the presence of long requests in the batch. A batch containing a 500-token request and 15 short requests has the short requests waiting for those 500 decode steps. Under static batching, this causes occasional enormous latency spikes at P99. Continuous batching reduces but does not eliminate this effect, because long requests still hold batch slots.

Always benchmark at multiple load levels, report P50, P95, and P99 latencies, and hold the system at each load level for at least five minutes before recording. Load spikes that last only a few seconds often look fine in benchmarks but cause cascading timeouts in production.