Continuous Batching: Optimizing LLM Inference Throughput

Large language model inference presents a unique challenge that sets it apart from nearly every other machine learning workload: requests arrive at different times, require different numbers of output tokens, and finish at unpredictable moments. Traditional batch processing, borrowed from training workflows where uniformity is the norm, handles this reality poorly. A batch must wait for its slowest member, leaving GPU resources idle while fast requests sit completed but unable to exit. The result is a system that wastes expensive compute cycles and creates frustrating delays when users expect snappy responses.

Think of static batching like a restaurant where a table of eight cannot leave until every single person has finished their meal. The first three diners who finish quickly just sit there, holding up the table, while the rest of the group takes their time. The restaurant cannot seat the next group until everyone leaves together, even though three slots have been effectively empty for minutes. This is exactly what happens to GPU compute slots under static batching: finished requests occupy space they no longer need, blocking new arrivals from starting.

Continuous batching solves this basic mismatch by rethinking how we group work together. Rather than treating a batch as a monolithic unit processed as a single group, continuous batching treats each decoding iteration as an opportunity to reshuffle which requests are actively using compute. Completed requests exit immediately, and waiting requests join without delay. The batch becomes a living, breathing set of concurrent streams rather than a frozen snapshot of work collected at a single moment in time.

The key insight is that autoregressive generation offers a natural scheduling boundary after every single token. Between any two decoding iterations, the model has completed one forward pass and is about to begin the next. At this boundary, we have complete freedom to change the composition of the batch, adding new requests and removing finished ones. Static batching ignores this flexibility entirely. Continuous batching exploits it fully.

This chapter explores why static batching creates inefficiency, how continuous batching addresses it through iteration-level scheduling, and the mechanisms for handling request completion and memory management in a production setting. The throughput gains from this approach are substantial, often doubling or tripling the requests per second a single GPU can handle. Understanding continuous batching is needed for when you deploy language models in production, as it has become the standard technique in modern serving frameworks including vLLM, TensorRT-LLM, and Text Generation Inference.

Batching multiple requests together is needed for efficient GPU utilization. Modern GPUs contain thousands of processing cores designed to execute operations in parallel, and feeding them work one request at a time leaves most of this capability unused. As we discussed in the KV Cache chapter, autoregressive generation produces one token per forward pass. Each forward pass involves substantial overhead: loading model weights from memory, executing attention computations, and applying the feed-forward network. Without batching, each forward pass processes a single request, leaving most of the GPU's parallel processing capability unused. Matrix multiplications that could operate on batch sizes of 32 or 64 instead operate on a single row, materially underutilizing the thousands of available cores.

The arithmetic here is stark. A modern A100 GPU can perform roughly 312 teraFLOPS of bfloat16 computation. A single-request forward pass through a 7B parameter model performs on the order of 14 billion floating-point operations. Batching 32 requests together multiplies useful work by 32 while the overhead of loading model weights from memory stays roughly constant. The same weight loading that dominates a batch-size-1 forward pass becomes negligible when amortized across 32 concurrent requests. This memory-bandwidth arithmetic is why practical throughput depends so heavily on batching.

Training workflows batch naturally because all samples pass through the same computation graph with identical sequence lengths after padding. Every example in a training batch starts together, processes together, and finishes together. This uniformity makes scheduling trivial. Inference is different in three necessary ways that break this simple model:

• Variable input lengths: requests arrive with prompts of different sizes. You might submit a 10-token question while another user gives a 2000-token document for summarization.
• Variable output lengths: Some responses are 10 tokens, others are 1000. A simple factual answer completes quickly, while a detailed explanation continues for many iterations.
• Asynchronous arrivals: Requests come in continuously, not in neat batches. Users submit queries at unpredictable times, and the system cannot wait indefinitely to form perfect batches.

These differences create a basic tension between batching efficiency and response latency. We want large batches to maximize GPU utilization, but we also want to return responses quickly without waiting for other requests. Static batching resolves this tension badly: it picks one batch and forces everything to conform to it. Continuous batching resolves it well: it adapts the batch to the actual work, iteration by iteration.

The simplest approach to batching inference requests is static batching, where we collect a fixed group of requests, process them together until all complete, and then start the next batch. This approach mirrors how training works and requires minimal changes to existing infrastructure.

Before looking at the mechanics, it helps to understand why engineers reached for static batching in the first place. Training code already handles fixed-size batches. Inference engines built on training frameworks naturally inherit this structure. The first LLM serving systems, deployed by research teams on hardware originally intended for training, simply reused the batch abstraction they knew. The problems with this choice only became apparent at scale when high-variance request lengths exposed just how much computation was wasted waiting for laggards.

In static batching, the system follows a straightforward process that prioritizes simplicity over flexibility. The core idea is to treat a batch as an indivisible unit that processes from start to finish without any changes to its composition. This makes implementation easy but creates significant inefficiencies when requests have different characteristics.

The static batching algorithm proceeds through five distinct phases:

1. Accumulate requests. Wait until a batch of size $B$ requests arrives, or a timeout expires. During this phase, incoming requests queue up while the system decides whether enough work has accumulated to justify starting a new batch.
2. Pad to uniform length. Pad all input sequences to match the longest input in the batch. Since matrix operations require uniform dimensions, shorter sequences receive padding tokens that consume compute but produce no useful output.
3. Process together. Run all requests through every decoding iteration. The entire batch moves in lockstep, with each request generating one token per forward pass.
4. Wait for completion. Continue until the longest output sequence finishes. Even if some requests complete after just a few tokens, they remain in the batch, consuming resources while creating only padding.
5. Return results. Send all responses back and start the next batch. Only after every request has finished can the system collect a new batch and begin again.

The analysis reveals the core problem with static batching: requests that finish early continue consuming compute resources. In our example, some requests only needed around 20 tokens while the batch waited for requests generating nearly 200 tokens. Those short requests sat idle for over 150 iterations, wasting computational resources that could have been serving other users.

To understand why this matters, consider what happens inside the GPU during those wasted iterations. The completed request still occupies a slot in every tensor operation. The attention mechanism still computes over its position. The feed-forward network still processes its hidden state. The output is discarded as padding, but the compute is very real. This is not idle waiting; it is inefficient computation that burns energy and time without creating anything useful.

This inefficiency compounds across several dimensions:

• Memory waste. Completed requests retain their KV cache entries until the batch finishes. As we covered in the KV Cache Memory chapter, each request's cache can consume gigabytes of GPU memory. A 7B parameter model might need 1-2 GB per request at full context length. Keeping completed requests' caches prevents new requests from starting, creating a bottleneck even when compute capacity exists.
• Latency inflation. Short requests experience artificially high latency because they must wait for longer siblings in their batch. A 20-token response that takes 100ms to generate might not be returned for 1000ms while waiting for a 200-token sibling. From the user's perspective, this feels like unexplained slowness, even though the system technically completed the request quickly.
• GPU underutilization. As requests complete, the effective batch size shrinks. A batch starting with 8 requests might have only 2 active requests for its final iterations, sharply reducing throughput. The GPU cores that could be serving new requests instead sit mostly idle, processing padding tokens for the few remaining active sequences.

The gray regions in this visualization stand for wasted computation: the GPU performs forward passes that produce padding tokens, while the finished requests cannot release their resources. The shorter the original request, the larger the fraction of its total batch time spent waiting. In a workload with high variance in output lengths, which is typical of real deployments, the average efficiency can easily drop to 20-30%.

Continuous batching, also called iteration-level scheduling or in-flight batching, takes a fundamentally different approach to managing concurrent requests. Instead of treating a batch as a fixed unit that lives and dies together, it treats each decoding iteration as an independent scheduling decision. This seemingly simple change materially improves both throughput and latency.

Think of continuous batching like a highway toll plaza where each booth independently processes cars as they arrive, rather than waiting for a predetermined convoy to form before opening. Cars enter, pay their toll, and leave the moment they are done, regardless of when the car behind them arrived. The plaza is always running at full capacity because new cars enter as soon as a booth clears. No convoy waits for its slowest member; every car moves at its own natural speed.

The move from static to continuous batching requires rethinking the basic unit of work. In static batching, the unit is a batch of requests that live together. In continuous batching, the unit is a single decoding iteration applied to whatever set of requests currently occupies the available slots. The batch is not a fixed group of requests but a window of concurrent work that shifts continuously as requests enter and exit.

The key insight behind continuous batching emerges from a careful examination of how autoregressive generation works. Each decoding step produces exactly one token per request. Between any two decoding iterations, the model completes one forward pass and begins the next. At this boundary, we have complete freedom to reorganize which requests participate in the next iteration. The model does not care whether the same requests continue; it only needs valid input tokens and their associated KV caches.

This observation reveals that the rigid batch structure of static batching is entirely artificial. There is no technical requirement that forces us to keep completed requests around. We imposed that constraint for implementation simplicity, not because the underlying computation demands it. The batch boundary is just a software construct, and we can move it wherever we want.

With this insight, continuous batching introduces three powerful capabilities:

• Remove completed requests. Immediately release slots when requests hit their end token. The moment a request generates its end-of-sequence marker, it exits the batch and its resources become available.
• Add new requests. Insert waiting requests into freed slots. New requests join the batch at whatever iteration happens to have space, rather than waiting for a complete batch turnover.
• Maintain continuous flow. Keep the batch saturated with active work. The goal is to maintain maximum occupancy at all times, treating the batch like a continuously flowing river rather than a series of discrete buckets.

This turns batch processing from a rigid structure into a dynamic queue where requests flow through the system at their natural pace. Short requests complete quickly and exit. Long requests continue as needed. New requests enter as soon as space permits. The GPU always has real work to do.

The continuous batching algorithm operates in a perpetual loop that never truly stops between batches. Instead of collecting requests, processing them, and collecting again, it maintains a persistent set of active requests that evolves iteration by iteration. Each cycle through the loop is a single forward pass through the model.

The algorithm follows these steps in each iteration:

1. Check for completions. After each iteration, identify requests that generated an end-of-sequence token or reached their maximum length. These requests have finished their work and no longer need to participate.
2. Evict completed requests. Remove finished requests from the active batch, freeing their slots and KV cache memory. This happens immediately, not at some future batch boundary.
3. Check waiting queue. Look for pending requests that arrived while the batch was processing. New requests accumulate in a queue whenever all slots are occupied.
4. Insert new requests. Fill empty slots with waiting requests, running their prefill phase to build initial KV caches. These new requests join alongside existing decode phase requests.
5. Execute iteration. Run one decoding step for all active requests. Every active request generates exactly one token.
6. Return completed results. Send finished responses back to clients immediately. Users receive their results the moment generation completes, not when some arbitrary batch finishes.

The loop never pauses to collect a new batch. The transition from one set of requests to another occurs within the normal iteration boundary. From the GPU's perspective, every iteration looks nearly identical: a batch of tokens arrives, a batch of logits comes out. Only the composition of the batch changes over time.

The high utilization and reduced iteration count show how continuous batching effectively saturates the GPU. By immediately filling slots vacated by completed requests, the system minimizes idle time and approaches the theoretical maximum throughput for the given workload. Notice that the total iterations is much closer to the theoretical minimum than what static batching achieved. The difference becomes even larger with realistic workloads that have higher variance in output lengths.

The power of continuous batching comes from making scheduling decisions at every iteration. Rather than committing to a fixed plan when a batch starts, the system reevaluates its choices thousands of times per second. This fine-grained control lets advanced policies beyond simple FIFO ordering and allows the system to adapt dynamically to changing conditions.

Iteration-level scheduling is a fundamentally different mental model from batch-level scheduling. In batch-level thinking, the scheduler plans who gets to run for the next several hundred milliseconds. In iteration-level thinking, the scheduler decides who runs for the next 10 to 50 milliseconds, then immediately reconsiders. This rapid reconsideration is what lets the system to react quickly to completions, memory pressure, and priority changes.

Think of iteration-level scheduling like air traffic control at a busy airport. Rather than assigning each plane to a runway at the start of the day and having it sit there regardless of what happens, air traffic controllers continuously decide which plane goes next. They adapt to delays and changing weather, including emergencies. A plane that taxied out for a 30-minute flight does not hold its runway slot while slower cross-country flights catch up. The runway is always occupied with the next available plane.

At each iteration boundary, the scheduler faces several interconnected decisions that determine both throughput and fairness. These decisions happen in microseconds and must balance competing objectives.

Which requests to evict: Besides naturally completed requests, the scheduler might preempt requests to free memory for higher-priority work. This creates interesting tradeoffs between fairness and throughput. A request that has already consumed significant resources is sunk compute cost, but continuing it also delays new requests. The scheduler must weigh these factors according to the system's priorities. Most production systems avoid preemption except under severe memory pressure, preferring instead to rely on natural completion to free slots.

Which waiting requests to admit: When multiple requests are waiting, the scheduler chooses which to start first. This decision materially impacts user-perceived latency and overall system efficiency. Options include:

• FIFO. First-come, first-served, which is simple and fair. Every request eventually gets its turn in the order it arrived. This approach gives predictable behavior and prevents starvation.
• Shortest Job First. Prioritize requests with shorter expected outputs to minimize average latency. Short requests complete quickly, improving user experience. However, this requires predicting output length, which is not always possible.
• Priority-based. Honor explicit priority levels attached to requests. Time-sensitive applications might receive preferential treatment. This approach requires clear policies about what constitutes priority and careful tuning to prevent starvation of low-priority requests.

How to handle memory pressure: When the KV cache approaches capacity, the scheduler must decide whether to wait for completions or preempt active requests. Waiting reduces throughput but preserves work in progress. Preemption frees resources immediately but wastes the computation already invested in the preempted request. Systems like vLLM use a "swap" mechanism that saves KV caches to CPU memory before preemption, letting resumption without full recomputation, though this adds latency.

The scheduling policy also determines how the system handles surges. When a sudden burst of requests arrives and the queue grows long, the scheduler must decide whether to admit many short requests quickly or maintain fair ordering. Most production systems prioritize fairness to avoid the situation where a user submitting a long job during a burst waits arbitrarily longer than someone who arrived moments later.

Continuous batching must handle requests in different phases of their lifecycle. As we discussed in the KV Cache chapter, generation has two distinct phases with fundamentally different computational characteristics.

Prefill phase: Process all input tokens in parallel to build the initial KV cache. This phase is compute-bound and benefits from large batches of tokens. The GPU performs dense matrix multiplications over potentially thousands of input tokens, fully utilizing its computational capacity. Memory bandwidth is not the bottleneck during prefill because the computation is so intensive. A prompt with 2000 tokens generates 2000 key-value pairs simultaneously in a single forward pass.

Decode phase: Generate tokens one at a time, reading from the KV cache. This phase is memory-bandwidth-bound and benefits from large batches of requests. Each decode step reads the entire KV cache but performs relatively little computation per byte loaded. The GPU spends most of its time waiting for memory operations to complete, not performing arithmetic. This is why batching many requests together in the decode phase is so important: each request reads a different part of memory in parallel, amortizing the memory latency across many concurrent streams.

These phases have different computational characteristics that create challenges when mixing them in the same batch. Prefill processes many tokens per request; decode processes one token per request. Mixing them creates load imbalance: prefill requests demand heavy compute while decode requests sit largely idle waiting for memory operations to complete. The compute and memory bandwidth resources are used differently, which makes it harder to fully utilize either.

Modern continuous batching systems often use chunked prefill to better balance the two phases and smooth out the computational load. Instead of processing the entire prompt in one iteration, which can cause latency spikes when long prompts arrive, long prompts are split into manageable chunks.

The problem that chunked prefill solves is sometimes called "prefill monopolization." When a request with a 4000-token prompt arrives, the prefill phase consumes the entire GPU for 10-15 milliseconds while thousands of queued decode steps for existing requests are blocked. From those existing requests' perspectives, the system has frozen. Their time-to-next-token spikes sharply, even though they have been running smoothly for hundreds of iterations.

The chunking strategy divides a long prompt into pieces that can be processed incrementally. This allows decode iterations for other requests to proceed between prefill chunks, maintaining responsiveness. A request with a 3000-token prompt might be split into six chunks of 500 tokens each, with each chunk processed in a separate iteration, interleaved with decode steps for existing requests.

Chunked prefill lets interleaving decode tokens between prefill chunks, reducing latency spikes when long prompts arrive. Without chunking, a 3000-token prompt would monopolize the GPU for a significant duration, blocking all decode operations for other requests. With chunking, decode operations proceed smoothly while the long prompt is processed incrementally. The cost is that the prefilling request takes longer to complete its context ingestion, but this tradeoff is almost always worth it in serving scenarios with multiple concurrent users.

Handling completed requests efficiently is important for continuous batching performance. The system must quickly detect completions, release resources, and fill the vacated slots. Any delay in this process reduces the benefits of continuous batching by leaving slots empty when work is available.

The completion handling pipeline runs at the boundary between every pair of iterations. It must execute fast enough to avoid becoming a bottleneck: if the completion check takes 5ms and the decode step takes 10ms, you have added 50% overhead. Production implementations use highly optimized routines, often scanning completion conditions directly on GPU tensors without transferring data to CPU.

A request completes when it generates a special end-of-sequence token or reaches a maximum length limit. The completion check happens after each decode step and must be fast enough to avoid becoming a bottleneck. In practice, this means simple integer comparisons that execute in nanoseconds.

The end-of-sequence token is a special marker that the model has learned to generate when it believes the response is complete. Different models use different token IDs for this purpose. GPT-family models typically use token ID 50256, while LLaMA models use token ID 2. The maximum length limit is a safety valve, preventing runaway generation that would consume resources to long sequences. Setting this limit requires balancing protection against runaway costs with flexibility for legitimately long responses.

When a request completes, the system must release several resources promptly to make room for new work. Each resource type has its own release mechanism and timing considerations.

• KV cache memory. The cached key-value tensors for all layers must be freed. With paged attention, as discussed in the Paged Attention chapter, this means returning memory blocks to the free pool. The blocks become immediately available for allocation to new requests, maximizing memory utilization.
• Batch slot. The position in the batch tensor becomes available for a new request. This slot is one concurrent generation stream, and freeing it allows the waiting queue to advance.
• Request metadata. Tracking structures for the completed request can be cleaned up. This includes timing information, token histories, and any other per-request state maintained by the scheduler.

The order of these release operations matters. KV cache memory should be freed before the new request attempts to allocate it. The batch slot assignment should complete before the next iteration begins. Careful sequencing ensures that resource counters remain consistent and that we never accidentally double-allocate memory.

Releasing the slot immediately returns blocks to the free pool, making them available for waiting requests in the very next iteration. This rapid turnover is what lets continuous batching to maintain high utilization. Notice that the freed blocks become available synchronously, so the very next call to _fill_batch can allocate them to a new request without any delay.

Unlike static batching, continuous batching can return completed responses immediately. Users don't wait for other requests in the batch, receiving results the moment generation finishes. This eliminates the latency inflation common in static batching and gives a more responsive experience.

The immediate delivery mechanism requires careful coordination between the generation loop and the response handling system. Completed results must be extracted and transmitted without blocking the main processing loop, often using asynchronous I/O or separate response threads. A naive implementation might pause the generation loop to send each response, but this would introduce jitter that accumulates across many requests. Production systems decouple response delivery from generation, queuing completed results for transmission by a separate thread or event loop.

Let's compare the throughput of static and continuous batching quantitatively. Understanding the mathematical relationship between these approaches helps explain why continuous batching delivers such significant improvements and under what conditions those improvements are largest.

The analysis that follows formalizes the intuition we built earlier: static batching allocates computational capacity based on the worst case in each batch, while continuous batching adapts to the actual workload token by token. The efficiency gap between these strategies grows with the variance of output lengths, which is exactly what characterizes real LLM workloads.

To understand the efficiency difference mathematically, we need to formalize what happens in each batching strategy. The key insight is that static batching allocates computational capacity based on the worst case, while continuous batching adapts to the actual workload.

For static batching with a fixed batch size $B$ containing requests with output lengths $L_1, L_2, \ldots, L_B$, the total computational time is determined by the longest request. Every request in the batch must wait for this slowest member, regardless of whether it has long since completed. The total computational capacity allocated, measured in token-slots (one slot processing one token), is:

where:

• $C_{\text{static}}$: the total number of processing slots consumed (iterations $\times$ batch size)
• $B$: the fixed batch size
• $L_i$: the output length of the $i$-th request
• $\max(L_1, \ldots, L_B)$: the length of the longest sequence in the batch

This formula captures the basic problem: we allocate capacity based on the maximum, not the sum. Every slot runs for as many iterations as the longest sequence demands, even if that slot has been creating padding for 90% of those iterations.

We can calculate the compute efficiency $\eta_{\text{static}}$ as the ratio of useful work to total capacity. Useful work is the actual tokens generated, while total capacity is the processing slots allocated:

where:

• $\eta_{\text{static}}$: the computational efficiency (between 0 and 1)
• $\sum_{i=1}^{B} L_i$: the total number of generated tokens (useful work)
• $B \times \max_{j} L_j$: the total capacity allocated during the batch's lifetime

Notice that efficiency depends heavily on the distribution of output lengths. If all requests have the same length, the numerator equals the denominator and efficiency is 100%. But if one request is much longer than the others, efficiency drops sharply.

To make this concrete, consider a batch of 8 requests where 7 need 20 tokens and 1 needs 200 tokens. We can work through the efficiency calculation step by step:

The system performs 1600 computational operations but only 340 of them produce useful output. That is 79% wasted compute, or a 21% efficiency rating. The single outlier request forces seven other requests to continue burning resources for 180 extra iterations each.

For continuous batching, the situation is fundamentally different. Assuming the request queue remains populated (perfect slot filling), the system avoids waiting for long requests. When a request completes, its slot immediately fills with a new request. The total iterations required approach the theoretical minimum:

where:

• $T_{\text{continuous}}$: the approximate total iterations needed
• $N$: the total number of requests in the workload
• $L_i$: the output length of the $i$-th request
• $B$: the maximum batch size (concurrency limit)
• $\sum_{i=1}^{N} L_i$: the total number of tokens to be generated across all requests

This formula shows that continuous batching distributes work evenly across the available batch slots. The total iterations is simply the total tokens divided by the parallelism factor. Doubling $B$ halves the time, and adding more requests to a non-empty queue barely changes the total time because they fill slots freed by completions.

We can derive the efficiency $\eta_{\text{continuous}}$ by comparing useful work to allocated capacity:

This shows that efficiency approaches 100% because the total capacity used matches the useful work. Every slot-iteration produces a real output token rather than padding. The approximation becomes exact when the queue is always full and requests are perfectly divisible across slots.

The practical implication is that continuous batching eliminates the sensitivity to output length distribution. Whether requests generate 10 tokens or 1000, the throughput stays near the theoretical maximum. Static batching is at the mercy of whoever happens to be in the same batch as you, while continuous batching processes everyone at their natural speed.

Let's run a more complete simulation comparing the two approaches across a realistic workload. Theory tells us continuous batching should win, but simulation reveals the magnitude of improvement and how it varies with workload characteristics.

The improvement from continuous batching grows with output length variance. When all requests generate similar lengths, static batching loses less to waiting because the maximum is close to the average. But real workloads have high variance: some users ask for single-sentence summaries while others request long document analyses. This variance makes continuous batching useful in practical deployments.

To cement the intuition, let's trace through a small concrete example comparing both strategies on the same five requests. This walkthrough shows exactly what happens at each iteration and why the results differ.

Suppose we have a maximum batch size of 3, and five requests arrive all at once with the following output lengths: Request A needs 4 tokens, Request B needs 2 tokens, Request C needs 6 tokens, Request D needs 3 tokens, and Request E needs 5 tokens.

Static batching: The system collects all five requests but the batch size is 3, so it forms two batches. Batch 1 contains Requests A, B, C. Batch 2 contains Requests D and E (padded to a batch of 2 since no more requests exist).

For Batch 1, the longest request is C with 6 tokens. All three requests run for 6 iterations:

• Iterations 1-2: A, B, C all active (B finishes at iteration 2 but stays in batch)
• Iterations 3-4: A, B (idle), C all running (A finishes at iteration 4 but stays in batch)
• Iterations 5-6: A (idle), B (idle), C all running

Total slot-iterations for Batch 1: $3 \times 6 = 18$. Useful tokens: $4 + 2 + 6 = 12$. Efficiency: $12/18 \approx 67\%$.

Batch 2 begins only after Batch 1 finishes. The longest of D (3 tokens) and E (5 tokens) is 5, so both run for 5 iterations.

Total slot-iterations for Batch 2: $2 \times 5 = 10$. Useful tokens: $3 + 5 = 8$. Efficiency: $8/10 = 80\%$.

Grand total: 28 slot-iterations, 20 useful tokens, overall efficiency $20/28 \approx 71\%$. Total wall-clock iterations: $6 + 5 = 11$.

Continuous batching: The system loads Requests A, B, C into the three available slots and begins immediately. Requests D and E wait in the queue.

• Iteration 1: A generates token 1, B generates token 1, C generates token 1. No completions. Queue: D, E.
• Iteration 2: A generates token 2, B generates token 2 (B is now complete), C generates token 2. B exits. D enters. Slot composition: A, C, D.
• Iteration 3: A generates token 3, C generates token 3, D generates token 1. No completions. Queue: E.
• Iteration 4: A generates token 4 (A complete), C generates token 4, D generates token 2. A exits. E enters. Slot composition: C, D, E.
• Iteration 5: C generates token 5, D generates token 3 (D complete), E generates token 1. D exits. Queue empty. Slot composition: C, E.
• Iteration 6: C generates token 6 (C complete), E generates token 2. C exits. Slot composition: E.
• Iteration 7: E generates token 3. No completions. Slot composition: E.
• Iteration 8: E generates token 4. No completions. Slot composition: E.
• Iteration 9: E generates token 5 (E complete). All done.

Total wall-clock iterations: 9. Total slot-iterations used: let's count. Iterations 1-4 had 3 active requests each (12), iteration 5 had 2 (2), iteration 6 had 2 (2), iterations 7-9 had 1 each (3). Total: $12 + 2 + 2 + 3 = 19$ slot-iterations for 20 useful tokens. Efficiency: $20/19$ which rounds to approximately 100% once you account for the single unavoidable idle slot-iteration when only E remained.

Static batching needed 11 wall-clock iterations and 28 slot-iterations. Continuous batching needed 9 wall-clock iterations and used only 19 slot-iterations. This is a modest example; with larger batches and higher variance, the gap grows substantially.

Building a production continuous batching system requires careful attention to several practical concerns that go beyond the basic algorithm. The theory is elegant, but the engineering details determine whether the system reaches its theoretical potential.

The main challenges fall into four areas: memory management, preemption handling, tensor operations, and monitoring. Each requires careful design. A system that gets the scheduling logic right but fails at memory management will hit out-of-memory errors under load. A system that handles memory well but lacks proper monitoring will be opaque when performance degrades.

Continuous batching works best when paired with paged attention, as covered in the Paged Attention chapter. Without it, memory fragmentation limits how many requests can be active simultaneously. Traditional contiguous allocation reserves maximum-length KV cache space for each request, even if most requests use far less. Paged attention allocates memory in small blocks on demand, letting much higher concurrency.

The interaction between continuous batching and paged memory is symbiotic. Continuous batching needs fine-grained memory release to quickly fill freed slots. Paged attention gives exactly this: when a request completes, its pages return to the free pool immediately, ready for allocation to incoming requests. The combination reaches both high throughput (through continuous batching) and high memory efficiency (through paged attention).

When memory pressure is high, the scheduler might need to preempt active requests to make room for higher-priority work. This involves saving the request's state, including the KV cache and generated tokens, for later resumption. Preemption is a powerful tool for managing resources but adds significant complexity.

The decision of which request to preempt involves tradeoffs. A policy must balance fairness against throughput while remaining practical to implement. Preempting a request that has generated many tokens wastes the computation invested in it. Preempting a request that has been preempted before may seem unfair. Different policies balance these concerns differently. vLLM's original implementation used a "beam search" style where preempted requests had their KV caches swapped to CPU DRAM, letting resumption without full recomputation when memory pressure eased. This swap-based approach preserves work in progress at the cost of CPU memory and PCIe bandwidth.

An important nuance is that preemption should be a last resort, not a routine operation. Systems that preempt frequently can enter a thrashing state where they spend more time managing preemption and resumption than doing useful generation. The right response to chronic memory pressure is usually to reduce the maximum batch size rather than to preempt aggressively.

For efficiency, token generation for all active requests should happen in a single batched forward pass. This requires careful tensor management to handle requests at different sequence positions. Unlike training, where all sequences in a batch are padded to the same length, continuous batching must handle sequences of varying lengths efficiently.

The key challenge is that different requests have different position embeddings and attention masks. Position embeddings tell the model where each token sits in its sequence. Attention masks prevent tokens from attending to future positions. Both vary by request, but both must be batched together for efficient GPU utilization.

The solution is to process just one token per request per forward pass during the decode phase. Each request contributes a single token to the batch input, and each token carries its own position identifier. The position IDs differ across requests in the batch because each request is at a different point in its generation, but the batch still processes as a single matrix operation.

The key insight is that despite requests being at different points in their generation, they can share a single forward pass because attention masks and position embeddings handle the differences. The input is just one token per request, but each token gets its own position embedding corresponding to its place in that request's sequence. The four requests above are at positions 50, 200, 15, and 500 in their respective sequences, yet they process together in the same matrix operations. This batching is what makes continuous batching efficient rather than just correct.

Tuning a continuous batching system requires understanding how several parameters interact. Higher throughput may increase latency or memory use, and reducing memory pressure can affect the other two measures. Each parameter affects multiple aspects of system behavior, so changes often have non-obvious second-order effects.

The key parameters for the continuous batching implementation are:

• max_batch_size: Maximum number of concurrent requests. Higher values improve throughput but require more memory for KV caches. The optimal value depends on model size, available GPU memory, and expected request characteristics. Setting this too high leads to out-of-memory errors; too low leaves GPU compute underutilized.
• block_size: Number of tokens per memory block in paged attention. Typically 16 or 32. Smaller blocks reduce internal fragmentation (wasted space within a block) but increase bookkeeping overhead. Larger blocks are simpler to manage but waste more memory for short sequences.
• chunk_size: Number of tokens processed per iteration during prefill. Helps balance latency and throughput. Smaller chunks give more responsive decode (less preemption of existing requests) but may reduce prefill efficiency due to increased per-iteration overhead.
• scheduler_policy: The ordering policy for the waiting queue (FIFO, SJF, priority). This affects fairness and average latency more than raw throughput.
• preemption_threshold: How full the memory pool must be before considering preemption. Setting this too low causes premature preemption; too high risks out-of-memory errors when a burst of long requests arrives.

Production continuous batching systems like vLLM, TensorRT-LLM, and Text Generation Inference report significant improvements over static batching. These systems combine continuous batching with paged attention, optimized kernels, and advanced scheduling to reach impressive throughput on commodity hardware.

The gains in production are often even larger than the simulations suggest, because real workloads have extreme variance. A production LLM serving endpoint might receive requests ranging from two-token completions to 4000-token document generations, all mixed together in the same serving pool. Static batching degrades catastrophically in this setting; continuous batching handles it naturally.

The throughput advantage of continuous batching compounds at larger batch sizes because GPU utilization stays high even as requests complete at different times. Static batching's throughput plateaus as the "waiting for slowest" overhead dominates, while continuous batching continues to scale. The latency curves tell an equally important story: continuous batching's latency grows slowly because each request's wait time is bounded by how long it takes to process its own tokens, not the tokens of batch-mates.

Continuous batching improves LLM serving but comes with challenges and tradeoffs. Its throughput and latency benefits require additional engineering.

The primary complexity lies in implementation. A continuous batching system requires tight integration between the scheduler, memory manager, and model execution engine. The scheduler must make split-second decisions about which requests to run, potentially thousands of times per second. Any inefficiency in this hot path directly reduces throughput. Debugging also becomes harder because requests can be preempted and later resumed while other requests run between them. Recreating issues requires capturing the exact sequence of scheduling decisions, which may be difficult in a highly dynamic system. A request that fails intermittently in production might have been scheduled next to a particularly long request or admitted just as memory pressure spiked; these interactions are difficult to replay in isolation.

Memory management presents another significant challenge. While paged attention solves fragmentation, the combination of variable-length requests and preemption creates complex allocation patterns. Memory pressure can cascade: preempting one request to admit another might itself require preemption of a third request, leading to difficult-to-predict behavior under load. Production systems need careful tuning of admission policies and preemption thresholds to avoid thrashing, where the system spends more time managing state transitions than generating tokens. The right setup depends heavily on the expected workload distribution, which can shift over time.

There are also fairness considerations that static batching never needed to address. Without careful policy design, long requests might be repeatedly preempted in favor of short ones, leading to starvation where a document-generation task waits indefinitely while the system processes an endless stream of brief queries. Conversely, prioritizing completion of long requests might inflate latency for short queries. Finding the right balance depends on the specific application's requirements and user expectations. Many production systems expose scheduler policies as setup, letting operators to tune the tradeoff for their use case.

A subtler limitation involves the mixed-phase problem. When prefill and decode requests share the same batch, neither phase gets optimal hardware utilization. Prefill benefits from large token batches that saturate compute; decode benefits from large request batches that saturate memory bandwidth. Mixing them creates an in-between state that is not ideal for either. Chunked prefill mitigates this by limiting how much prefill work can occur in any single iteration, but it does not eliminate the basic tension. Some systems address this by maintaining separate prefill and decode "pools" that coordinate to share GPU time, adding further scheduling complexity.

Despite these challenges, continuous batching has fundamentally changed LLM deployment economics. Systems that previously needed multiple GPUs to serve moderate traffic can now handle the same load on a single device. This efficiency improvement makes LLMs practical for a much wider range of applications and organizations. The technique has become standard in production serving frameworks, and understanding its principles is needed for anyone deploying language models at scale. The Orca paper that introduced the idea reported up to 36.9x throughput improvements, and production deployments regularly reach 2-5x improvements over equivalent static batching configurations.

As we'll explore in the next chapter on Inference Serving, continuous batching is just one component of a complete serving system. Real deployments must also handle request routing, load balancing, model replication, and failure recovery, all while maintaining the low latency users expect.

Continuous batching turns LLM inference from rigid, inefficient batch processing into a dynamic, high-utilization system. The main ideas we covered include:

• Static batching inefficiency. Traditional batching forces all requests to wait for the slowest, wasting compute on completed requests and inflating latencies. Efficiency can drop to 20% or below for workloads with high output length variance.
• Iteration-level scheduling. Treating each decode step as a scheduling opportunity allows immediate eviction of completed requests and admission of waiting ones. The batch composition changes continuously rather than at fixed batch boundaries.
• Phase handling. Managing the distinct prefill and decode phases, potentially with chunked prefill, balances compute characteristics across the batch and prevents long prompts from monopolizing the GPU.
• Resource management. Tight integration with KV cache memory management, including paged attention, maximizes the number of concurrent requests by returning memory to the pool immediately on completion.
• Theoretical efficiency. Continuous batching approaches 100% compute efficiency because total capacity used matches total useful work, independent of output length variance.
• Throughput gains. Real-world improvements of 2-5x throughput are common, with better latency characteristics especially for short requests.

Continuous batching is now standard practice for production LLM serving. Combined with the memory optimizations from paged attention and the speculative decoding techniques we covered earlier, it lets serving LLMs at scale with reasonable hardware costs.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about continuous batching.