Continuous Batching: Optimizing LLM Inference Throughput › The Batching Problem in LLM Inference

Batching multiple requests together is needed for efficient GPU utilization. Modern GPUs contain thousands of processing cores designed to execute operations in parallel, and feeding them work one request at a time leaves most of this capability unused. As we discussed in the KV Cache chapter, autoregressive generation produces one token per forward pass. Each forward pass involves substantial overhead: loading model weights from memory, executing attention computations, and applying the feed-forward network. Without batching, each forward pass processes a single request, leaving most of the GPU's parallel processing capability unused. Matrix multiplications that could operate on batch sizes of 32 or 64 instead operate on a single row, materially underutilizing the thousands of available cores.

The arithmetic here is stark. A modern A100 GPU can perform roughly 312 teraFLOPS of bfloat16 computation. A single-request forward pass through a 7B parameter model performs on the order of 14 billion floating-point operations. Batching 32 requests together multiplies useful work by 32 while the overhead of loading model weights from memory stays roughly constant. The same weight loading that dominates a batch-size-1 forward pass becomes negligible when amortized across 32 concurrent requests. This memory-bandwidth arithmetic is why practical throughput depends so heavily on batching.

Training workflows batch naturally because all samples pass through the same computation graph with identical sequence lengths after padding. Every example in a training batch starts together, processes together, and finishes together. This uniformity makes scheduling trivial. Inference is different in three necessary ways that break this simple model:

• Variable input lengths: requests arrive with prompts of different sizes. You might submit a 10-token question while another user gives a 2000-token document for summarization.
• Variable output lengths: Some responses are 10 tokens, others are 1000. A simple factual answer completes quickly, while a detailed explanation continues for many iterations.
• Asynchronous arrivals: Requests come in continuously, not in neat batches. Users submit queries at unpredictable times, and the system cannot wait indefinitely to form perfect batches.

These differences create a basic tension between batching efficiency and response latency. We want large batches to maximize GPU utilization, but we also want to return responses quickly without waiting for other requests. Static batching resolves this tension badly: it picks one batch and forces everything to conform to it. Continuous batching resolves it well: it adapts the batch to the actual work, iteration by iteration.