Continuous Batching: Optimizing LLM Inference Throughput › Limitations and Impact

Continuous batching improves LLM serving but comes with challenges and tradeoffs. Its throughput and latency benefits require additional engineering.

The primary complexity lies in implementation. A continuous batching system requires tight integration between the scheduler, memory manager, and model execution engine. The scheduler must make split-second decisions about which requests to run, potentially thousands of times per second. Any inefficiency in this hot path directly reduces throughput. Debugging also becomes harder because requests can be preempted and later resumed while other requests run between them. Recreating issues requires capturing the exact sequence of scheduling decisions, which may be difficult in a highly dynamic system. A request that fails intermittently in production might have been scheduled next to a particularly long request or admitted just as memory pressure spiked; these interactions are difficult to replay in isolation.

Memory management presents another significant challenge. While paged attention solves fragmentation, the combination of variable-length requests and preemption creates complex allocation patterns. Memory pressure can cascade: preempting one request to admit another might itself require preemption of a third request, leading to difficult-to-predict behavior under load. Production systems need careful tuning of admission policies and preemption thresholds to avoid thrashing, where the system spends more time managing state transitions than generating tokens. The right setup depends heavily on the expected workload distribution, which can shift over time.

There are also fairness considerations that static batching never needed to address. Without careful policy design, long requests might be repeatedly preempted in favor of short ones, leading to starvation where a document-generation task waits indefinitely while the system processes an endless stream of brief queries. Conversely, prioritizing completion of long requests might inflate latency for short queries. Finding the right balance depends on the specific application's requirements and user expectations. Many production systems expose scheduler policies as setup, letting operators to tune the tradeoff for their use case.

A subtler limitation involves the mixed-phase problem. When prefill and decode requests share the same batch, neither phase gets optimal hardware utilization. Prefill benefits from large token batches that saturate compute; decode benefits from large request batches that saturate memory bandwidth. Mixing them creates an in-between state that is not ideal for either. Chunked prefill mitigates this by limiting how much prefill work can occur in any single iteration, but it does not eliminate the basic tension. Some systems address this by maintaining separate prefill and decode "pools" that coordinate to share GPU time, adding further scheduling complexity.

Despite these challenges, continuous batching has fundamentally changed LLM deployment economics. Systems that previously needed multiple GPUs to serve moderate traffic can now handle the same load on a single device. This efficiency improvement makes LLMs practical for a much wider range of applications and organizations. The technique has become standard in production serving frameworks, and understanding its principles is needed for anyone deploying language models at scale. The Orca paper that introduced the idea reported up to 36.9x throughput improvements, and production deployments regularly reach 2-5x improvements over equivalent static batching configurations.

As we'll explore in the next chapter on Inference Serving, continuous batching is just one component of a complete serving system. Real deployments must also handle request routing, load balancing, model replication, and failure recovery, all while maintaining the low latency users expect.