Throughput Optimization › Static Batching and Why It Falls Short

The most intuitive approach to improving throughput is batching: group multiple requests together and process them as a unit. A single matrix multiplication over a batch of 32 input sequences is much faster than 32 separate matrix multiplications, because modern GPUs excel at wide, parallel operations. This observation is correct as far as it goes, but language model generation introduces complications that make naive batching surprisingly inefficient.

In static batching, you accumulate requests until you have a full batch (or a timeout expires), then process all of them together. Every request in the batch enters the model at the same time and exits at the same time. The batch is treated as an atomic unit from the moment it is assembled to the moment the last token is generated.

The batch size $B$ directly controls throughput. Each forward pass through the model costs roughly $C$ milliseconds of GPU time. A batch of $B$ requests costs approximately $C$ milliseconds and produces $B$ responses, so throughput scales as:

$$\text{throughput} = \frac{B}{C}$$

where:

• $B$: the batch size (number of requests processed together)
• $C$: the time in milliseconds for one forward pass over the batch

Doubling $B$ doubles throughput as long as $C$ stays constant, which is approximately true when the model is compute-bound and GPU memory can fit the larger batch. This observation motivated the development of batching in classical deep learning, and it holds reasonably well for single-pass inference tasks like image classification.

For language model generation, however, the formula is deceptive. The forward pass is not a single operation: it is $T$ sequential decode steps, where $T$ is the output length. And the output length is not fixed across requests. This is where static batching breaks down.

Language model generation introduces a critical complication: requests finish at different times. One user asks for a one-sentence summary (20 tokens), another asks for a detailed essay (500 tokens). In static batching, all requests in the batch must finish before any response is returned.

Consider the GPU computation from step 21 onwards when the short request is done. The GPU is generating tokens 21 through 500 for the long request. The short request's batch slot is occupied but producing nothing useful. No new request can enter the batch to fill that slot. The GPU is running decode steps, but only one of the 32 batch slots is doing productive work.

The idle slot problem compounds with batch size. In a batch of 32 requests with a bimodal length distribution (half short at 20 tokens, half long at 500 tokens), after step 20 you have 16 occupied slots and 16 idle slots. You are running at half efficiency for 480 steps. The waste grows with the ratio between the longest and shortest requests.

Worse, the memory occupied by the completed request's KV cache cannot be freed until the batch finishes, limiting how much memory is available for new requests. The KV cache for those 16 finished requests is still allocated, preventing the system from admitting new work even though the slots are logically idle.

Variable-length inputs create a second problem. To batch sequences together, you must pad shorter sequences to match the longest sequence in the batch. If your batch contains one 500-token request and 31 one-sentence requests, those 31 short requests are padded to 500 tokens. The attention mechanism processes all 500 positions for every request, even the empty padding positions.

This waste is not merely theoretical. Consider the attention computation for one layer: it processes a sequence length of 500 for every request. For the 31 padded requests, positions 20-500 are padding and their contribution to the output is zeroed out by the attention mask. The GPU does the arithmetic anyway: loading weights, computing queries, keys, and values, running softmax, applying the mask. All of that computation is discarded.

Depending on the length distribution of your workload, padding waste can reduce effective GPU utilization by 30-70%. You are doing real computation on fake data. For services with highly variable request lengths, which is nearly every production chat or completion service, static batching's padding overhead is severe enough to eliminate most of the throughput benefit of batching at all.

There are partial mitigations: bucketing requests by length and forming batches within length buckets, or using attention implementations that skip padding positions. But these are workarounds for a fundamental architectural mismatch. The real solution is continuous batching.