Continuous Batching: Optimizing LLM Inference Throughput › Static Batching
Language AI HandbookAll chapters Continuous Batching: Optimizing LLM Inference Throughput › Static BatchingOne section · Sections in this chapter3 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The simplest approach to batching inference requests is static batching, where we collect a fixed group of requests, process them together until all complete, and then start the next batch. This approach mirrors how training works and requires minimal changes to existing infrastructure.
Before looking at the mechanics, it helps to understand why engineers reached for static batching in the first place. Training code already handles fixed-size batches. Inference engines built on training frameworks naturally inherit this structure. The first LLM serving systems, deployed by research teams on hardware originally intended for training, simply reused the batch abstraction they knew. The problems with this choice only became apparent at scale when high-variance request lengths exposed just how much computation was wasted waiting for laggards.
How Static Batching Works
In static batching, the system follows a straightforward process that prioritizes simplicity over flexibility. The core idea is to treat a batch as an indivisible unit that processes from start to finish without any changes to its composition. This makes implementation easy but creates significant inefficiencies when requests have different characteristics.
The static batching algorithm proceeds through five distinct phases:
- Accumulate requests. Wait until a batch of size requests arrives, or a timeout expires. During this phase, incoming requests queue up while the system decides whether enough work has accumulated to justify starting a new batch.
- Pad to uniform length. Pad all input sequences to match the longest input in the batch. Since matrix operations require uniform dimensions, shorter sequences receive padding tokens that consume compute but produce no useful output.
- Process together. Run all requests through every decoding iteration. The entire batch moves in lockstep, with each request generating one token per forward pass.
- Wait for completion. Continue until the longest output sequence finishes. Even if some requests complete after just a few tokens, they remain in the batch, consuming resources while creating only padding.
- Return results. Send all responses back and start the next batch. Only after every request has finished can the system collect a new batch and begin again.
from dataclasses import dataclass
from typing import List
@dataclass
class Request:
"""Simulated inference request."""
id: int
input_length: int
output_length: int # How many tokens this request will generate
def simulate_static_batch(requests: List[Request]) -> dict:
"""Simulate static batching behavior."""
if not requests:
return {
"total_iterations": 0,
"wasted_iterations": 0,
"max_output": 0,
"useful_iterations": 0,
"efficiency": 0,
}
# All requests must wait for the longest output
max_output = max(r.output_length for r in requests)
# Calculate wasted computation
total_iterations = len(requests) * max_output
useful_iterations = sum(r.output_length for r in requests)
wasted_iterations = total_iterations - useful_iterations
return {
"max_output": max_output,
"total_iterations": total_iterations,
"useful_iterations": useful_iterations,
"wasted_iterations": wasted_iterations,
"efficiency": useful_iterations / total_iterations,
}Let's see how static batching performs with requests of varying output lengths. This simulation shows the core problem: when requests differ materially in how many tokens they need to generate, the short requests end up waiting for the long ones.
import numpy as np
# Create a batch with highly variable output lengths
static_batch = [
Request(id=i, input_length=50, output_length=np.random.randint(10, 200))
for i in range(8)
]
# Show the output lengths
output_lengths = [r.output_length for r in static_batch]Static Batch Request Output Lengths: Request 0: 112 tokens ██████████████████████ Request 1: 189 tokens █████████████████████████████████████ Request 2: 102 tokens ████████████████████ Request 3: 24 tokens ████ Request 4: 116 tokens ███████████████████████ Request 5: 81 tokens ████████████████ Request 6: 198 tokens ███████████████████████████████████████ Request 7: 30 tokens ██████
static_results = simulate_static_batch(static_batch)Static Batching Analysis: Maximum output length: 198 tokens Total iterations computed: 1584 Useful iterations: 852 Wasted iterations: 732 Compute efficiency: 53.8%
The Efficiency Problem
The analysis reveals the core problem with static batching: requests that finish early continue consuming compute resources. In our example, some requests only needed around 20 tokens while the batch waited for requests generating nearly 200 tokens. Those short requests sat idle for over 150 iterations, wasting computational resources that could have been serving other users.
To understand why this matters, consider what happens inside the GPU during those wasted iterations. The completed request still occupies a slot in every tensor operation. The attention mechanism still computes over its position. The feed-forward network still processes its hidden state. The output is discarded as padding, but the compute is very real. This is not idle waiting; it is inefficient computation that burns energy and time without creating anything useful.
This inefficiency compounds across several dimensions:
- Memory waste. Completed requests retain their KV cache entries until the batch finishes. As we covered in the KV Cache Memory chapter, each request's cache can consume gigabytes of GPU memory. A 7B parameter model might need 1-2 GB per request at full context length. Keeping completed requests' caches prevents new requests from starting, creating a bottleneck even when compute capacity exists.
- Latency inflation. Short requests experience artificially high latency because they must wait for longer siblings in their batch. A 20-token response that takes 100ms to generate might not be returned for 1000ms while waiting for a 200-token sibling. From the user's perspective, this feels like unexplained slowness, even though the system technically completed the request quickly.
- GPU underutilization. As requests complete, the effective batch size shrinks. A batch starting with 8 requests might have only 2 active requests for its final iterations, sharply reducing throughput. The GPU cores that could be serving new requests instead sit mostly idle, processing padding tokens for the few remaining active sequences.
The gray regions in this visualization stand for wasted computation: the GPU performs forward passes that produce padding tokens, while the finished requests cannot release their resources. The shorter the original request, the larger the fraction of its total batch time spent waiting. In a workload with high variance in output lengths, which is typical of real deployments, the average efficiency can easily drop to 20-30%.