Latency Optimization › Batching Strategies

Batching is the primary technique for improving GPU utilization during decode. The core idea is simple: if the GPU must read all model weights to decode one token, it might as well decode a token for many requests simultaneously. The memory bandwidth cost is roughly the same, while the compute throughput increases linearly with batch size.

Understanding the batching strategies available, and their trade-offs, is essential because the choice of batching strategy determines both throughput and latency behavior under realistic serving conditions. No single strategy is optimal for all workloads; the best choice depends on the distribution of request lengths and the latency SLOs you need to meet.

The simplest approach collects a fixed set of requests, runs them all together until all sequences are complete, and then collects the next batch. The problem is that sequences in a batch complete at different times. When the first sequence finishes, its slot in the batch sits idle until all other sequences finish. This padding waste is severe when sequence lengths vary widely.

To see just how bad this can be, consider a batch of eight requests where seven generate 50 tokens and one generates 400 tokens. The batch completes only when the 400-token sequence finishes. During the final 350 decode steps, seven of the eight batch slots are doing no useful work. GPU utilization for those steps is 12.5%. This is not a pathological edge case; real-world request length distributions are highly skewed, with the majority of requests being short and a long tail of much longer ones.

Static batching does have one advantage: simplicity. The entire batch is processed with a single fixed-size matrix operation at each decode step, which maps cleanly to standard CUDA kernels without requiring custom scatter-gather logic. For workloads with very uniform sequence lengths (all requests generating exactly $N$ tokens), static batching wastes no time and achieves high throughput. In practice, however, this uniformity rarely holds outside of benchmarking scenarios.

Continuous batching (also called dynamic batching or iteration-level scheduling) eliminates padding waste by inserting new requests at the level of individual decode iterations. When one sequence in a batch completes, the scheduler immediately fills its slot with a new request. The batch composition changes from step to step, but the GPU always runs at the target batch size.

This is now the standard approach in production systems. vLLM, TGI (Hugging Face Text Generation Inference), and NVIDIA Triton Inference Server all implement variants of continuous batching. The throughput improvement over static batching on realistic workloads is often 10-50x because the GPU stays busy rather than waiting for stragglers.

The key implementation insight is that every decode step is structurally identical: the model reads its weights, attends over cached keys and values, and produces a logit distribution for the next token. The fact that different sequences in the batch are at different positions in their respective generations does not change this structure. Each sequence simply provides a different query vector to the attention operation and reads from a different slice of the KV cache. With paged attention managing those cache slices, combining sequences at arbitrary positions in their generations is straightforward.

One important consideration for continuous batching is how to handle the prefill step for new requests that join the batch mid-iteration. Prefill is a different operation from decode: it processes many tokens at once in a single large matrix multiplication. Mixing prefill work with in-flight decode work in the same GPU kernel is architecturally complex. Most serving systems handle this by interleaving: one iteration computes the prefill for a newly arrived request, and the next iteration includes that request in the decode batch. This introduces one iteration of extra latency for new requests but keeps the implementation clean.

The visualization makes the inefficiency of static batching concrete. When the longest sequence in a batch is eight times the shortest, static batching achieves only about 12% utilization during the final steps. Continuous batching achieves near-full utilization by replacing completed requests immediately.

One tension in continuous batching is that long prefill operations can block decode steps. When a new request with a 4,000-token prompt joins the system, its prefill computation can take hundreds of milliseconds, delaying all in-flight decode requests and spiking TTFT for other users.

Chunked prefill addresses this by breaking large prompts into smaller chunks (for example, 512 tokens at a time) and interleaving them with decode steps. The prefill of a single long request is spread across many iteration steps, allowing decode requests to progress in between. This trades a modest increase in per-request TTFT (because the prefill now takes multiple iterations instead of one) for much better TTFT consistency across the system as a whole.

The chunk size is a tunable parameter. Smaller chunks reduce head-of-line blocking but increase the scheduling overhead of managing partially prefilled requests. Larger chunks reduce overhead but increase the worst-case TTFT spike for decode requests running in parallel. Production systems typically tune this parameter empirically based on observed workload characteristics.

Chunked prefill also interacts with the KV cache in an interesting way. A partially prefilled request consumes KV cache pages for the tokens processed so far, but the attention operation for those tokens is not yet useful for generation. The serving system must track the boundary between prefilled and not-yet-prefilled positions and ensure that attention is only computed over the prefilled portion until the full prompt is processed.