Continuous Batching: Optimizing LLM Inference Throughput › Continuous Batching
Language AI HandbookAll chapters Continuous Batching: Optimizing LLM Inference Throughput › Continuous BatchingOne section · Sections in this chapter4 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Continuous batching, also called iteration-level scheduling or in-flight batching, takes a fundamentally different approach to managing concurrent requests. Instead of treating a batch as a fixed unit that lives and dies together, it treats each decoding iteration as an independent scheduling decision. This seemingly simple change materially improves both throughput and latency.
Think of continuous batching like a highway toll plaza where each booth independently processes cars as they arrive, rather than waiting for a predetermined convoy to form before opening. Cars enter, pay their toll, and leave the moment they are done, regardless of when the car behind them arrived. The plaza is always running at full capacity because new cars enter as soon as a booth clears. No convoy waits for its slowest member; every car moves at its own natural speed.
The move from static to continuous batching requires rethinking the basic unit of work. In static batching, the unit is a batch of requests that live together. In continuous batching, the unit is a single decoding iteration applied to whatever set of requests currently occupies the available slots. The batch is not a fixed group of requests but a window of concurrent work that shifts continuously as requests enter and exit.
The Core Insight
The key insight behind continuous batching emerges from a careful examination of how autoregressive generation works. Each decoding step produces exactly one token per request. Between any two decoding iterations, the model completes one forward pass and begins the next. At this boundary, we have complete freedom to reorganize which requests participate in the next iteration. The model does not care whether the same requests continue; it only needs valid input tokens and their associated KV caches.
This observation reveals that the rigid batch structure of static batching is entirely artificial. There is no technical requirement that forces us to keep completed requests around. We imposed that constraint for implementation simplicity, not because the underlying computation demands it. The batch boundary is just a software construct, and we can move it wherever we want.
With this insight, continuous batching introduces three powerful capabilities:
- Remove completed requests. Immediately release slots when requests hit their end token. The moment a request generates its end-of-sequence marker, it exits the batch and its resources become available.
- Add new requests. Insert waiting requests into freed slots. New requests join the batch at whatever iteration happens to have space, rather than waiting for a complete batch turnover.
- Maintain continuous flow. Keep the batch saturated with active work. The goal is to maintain maximum occupancy at all times, treating the batch like a continuously flowing river rather than a series of discrete buckets.
This turns batch processing from a rigid structure into a dynamic queue where requests flow through the system at their natural pace. Short requests complete quickly and exit. Long requests continue as needed. New requests enter as soon as space permits. The GPU always has real work to do.
How Continuous Batching Works
The continuous batching algorithm operates in a perpetual loop that never truly stops between batches. Instead of collecting requests, processing them, and collecting again, it maintains a persistent set of active requests that evolves iteration by iteration. Each cycle through the loop is a single forward pass through the model.
The algorithm follows these steps in each iteration:
- Check for completions. After each iteration, identify requests that generated an end-of-sequence token or reached their maximum length. These requests have finished their work and no longer need to participate.
- Evict completed requests. Remove finished requests from the active batch, freeing their slots and KV cache memory. This happens immediately, not at some future batch boundary.
- Check waiting queue. Look for pending requests that arrived while the batch was processing. New requests accumulate in a queue whenever all slots are occupied.
- Insert new requests. Fill empty slots with waiting requests, running their prefill phase to build initial KV caches. These new requests join alongside existing decode phase requests.
- Execute iteration. Run one decoding step for all active requests. Every active request generates exactly one token.
- Return completed results. Send finished responses back to clients immediately. Users receive their results the moment generation completes, not when some arbitrary batch finishes.
The loop never pauses to collect a new batch. The transition from one set of requests to another occurs within the normal iteration boundary. From the GPU's perspective, every iteration looks nearly identical: a batch of tokens arrives, a batch of logits comes out. Only the composition of the batch changes over time.
from collections import deque
@dataclass
class ContinuousRequest:
"""Request with tracking for continuous batching."""
id: int
input_length: int
output_length: int # Total tokens to generate
tokens_generated: int = 0
start_iteration: int = 0
end_iteration: int = 0
@property
def is_complete(self) -> bool:
return self.tokens_generated >= self.output_length
class ContinuousBatchSimulator:
"""Simulates continuous batching behavior."""
def __init__(self, max_batch_size: int = 8):
self.max_batch_size = max_batch_size
self.active_batch: List[ContinuousRequest] = []
self.waiting_queue: deque = deque()
self.completed: List[ContinuousRequest] = []
self.iteration = 0
self.utilization_history = []
def submit_request(self, request: ContinuousRequest):
"""Add a request to the waiting queue."""
self.waiting_queue.append(request)
def _fill_batch(self):
"""Fill empty slots with waiting requests."""
while (
len(self.active_batch) < self.max_batch_size and self.waiting_queue
):
request = self.waiting_queue.popleft()
request.start_iteration = self.iteration
self.active_batch.append(request)
def _process_completions(self):
"""Remove completed requests from the batch."""
still_active = []
for request in self.active_batch:
if request.is_complete:
request.end_iteration = self.iteration
self.completed.append(request)
else:
still_active.append(request)
self.active_batch = still_active
def step(self):
"""Execute one iteration of the continuous batch."""
# First, fill any empty slots
self._fill_batch()
if not self.active_batch:
return False # Nothing to process
# Record utilization
self.utilization_history.append(
len(self.active_batch) / self.max_batch_size
)
# Generate one token for each active request
for request in self.active_batch:
request.tokens_generated += 1
self.iteration += 1
# Check for completions
self._process_completions()
return True
def run_until_complete(self):
"""Run until all requests are processed."""
while self.active_batch or self.waiting_queue:
self.step()Let's simulate the same requests with continuous batching to see the difference in practice:
# Convert our static batch to continuous batch format, then add extra requests
# so the batch (size 8) is always full and slot reuse is visible in the timeline
extra_requests = [
Request(id=i + 8, input_length=50, output_length=np.random.randint(10, 200))
for i in range(8)
]
all_requests = static_batch + extra_requests
continuous_requests = [
ContinuousRequest(
id=r.id, input_length=r.input_length, output_length=r.output_length
)
for r in all_requests
]
# Run continuous batching simulation with 16 requests and batch_size=8
# so the second half must wait and slots are visibly reused as the first
# requests finish at different times
simulator = ContinuousBatchSimulator(max_batch_size=8)
for request in continuous_requests:
simulator.submit_request(request)
simulator.run_until_complete()Continuous Batching Results: Total iterations: 322 Average utilization: 69.3% Theoretical minimum iterations: 223
The high utilization and reduced iteration count show how continuous batching effectively saturates the GPU. By immediately filling slots vacated by completed requests, the system minimizes idle time and approaches the theoretical maximum throughput for the given workload. Notice that the total iterations is much closer to the theoretical minimum than what static batching achieved. The difference becomes even larger with realistic workloads that have higher variance in output lengths.