Throughput Optimization › Implementation: Measuring and Optimizing Throughput
Language AI HandbookAll chapters Throughput Optimization › Implementation: Measuring and Optimizing ThroughputOne section · Sections in this chapter8 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us build a practical throughput measurement and optimization toolkit. We will simulate a serving system, measure throughput under different conditions, and visualize the impact of key decisions.
Simulating a Request Workload
We start by modeling realistic request distributions. Production workloads typically show a mix of short and long requests with a roughly log-normal length distribution. A few very long requests coexist with a large volume of short requests, and the tail is heavier than a normal distribution would suggest.
@dataclass
class Request:
request_id: int
prompt_tokens: int
max_output_tokens: int
arrival_time: float = 0.0
completion_time: float = 0.0
def generate_workload(
n_requests: int,
mean_prompt_tokens: int = 256,
mean_output_tokens: int = 128,
rng_seed: int = 42,
) -> List[Request]:
"""Generate a synthetic workload with log-normal length distributions."""
rng = np.random.default_rng(rng_seed)
# Log-normal distributions typical of production LLM workloads
prompt_lengths = (
rng.lognormal(
mean=np.log(mean_prompt_tokens), sigma=0.8, size=n_requests
)
.clip(10, 2048)
.astype(int)
)
output_lengths = (
rng.lognormal(
mean=np.log(mean_output_tokens), sigma=1.0, size=n_requests
)
.clip(5, 512)
.astype(int)
)
# Poisson arrival process
inter_arrivals = rng.exponential(
scale=1.0 / 10.0, size=n_requests
) # 10 req/s
arrival_times = np.cumsum(inter_arrivals)
return [
Request(i, int(p), int(o), float(a))
for i, (p, o, a) in enumerate(
zip(prompt_lengths, output_lengths, arrival_times)
)
]Generated workload: 500 requests Prompt lengths -- mean: 339, median: 256, p95: 923 Output lengths -- mean: 174, median: 128, p95: 512 Total tokens to generate: 87,172
The workload reflects the log-normal shape typical in production: most requests are short, but a long tail of complex requests generates the majority of total tokens. The P95 output length is substantially higher than the mean, which is precisely what makes continuous batching so valuable. Static batching at this workload would have most batch slots sitting idle while the long requests complete.
Simulating a Serving System
Next we build a simple serving simulator that can model both static and continuous batching. The simulator tracks wall-clock simulation time, prefill costs, and per-request completion times.
@dataclass
class ServingStats:
total_requests: int = 0
total_tokens_generated: int = 0
total_time_seconds: float = 0.0
latencies_ms: List[float] = field(default_factory=list)
@property
def request_throughput(self) -> float:
return (
self.total_requests / self.total_time_seconds
if self.total_time_seconds > 0
else 0.0
)
@property
def token_throughput(self) -> float:
return (
self.total_tokens_generated / self.total_time_seconds
if self.total_time_seconds > 0
else 0.0
)
@property
def p50_latency_ms(self) -> float:
return (
float(np.percentile(self.latencies_ms, 50))
if self.latencies_ms
else 0.0
)
@property
def p99_latency_ms(self) -> float:
return (
float(np.percentile(self.latencies_ms, 99))
if self.latencies_ms
else 0.0
)
def simulate_serving(
requests: List[Request],
batch_size: int,
ms_per_decode_step: float = 5.0,
ms_per_prefill_token: float = 0.1,
use_continuous_batching: bool = True,
) -> ServingStats:
"""
Simulate LLM serving with either static or continuous batching.
ms_per_decode_step: milliseconds per single generation step (all active requests advance one token)
ms_per_prefill_token: milliseconds to prefill one token in the prompt
"""
stats = ServingStats()
queue = list(requests)
queue.sort(key=lambda r: r.arrival_time)
current_time_ms = 0.0
active: List[Request] = []
remaining_tokens: List[int] = []
completed_requests: List[Request] = []
queue_idx = 0
while queue_idx < len(queue) or active:
# Prefill: admit new requests up to batch_size
while len(active) < batch_size and queue_idx < len(queue):
req = queue[queue_idx]
# Prefill cost
prefill_cost = req.prompt_tokens * ms_per_prefill_token
current_time_ms += prefill_cost
active.append(req)
remaining_tokens.append(req.max_output_tokens)
queue_idx += 1
if not active:
# Idle: jump to next arrival
if queue_idx < len(queue):
current_time_ms = max(
current_time_ms, queue[queue_idx].arrival_time * 1000
)
continue
if use_continuous_batching:
# One decode step: advance all active requests by 1 token
current_time_ms += ms_per_decode_step
new_active = []
new_remaining = []
for req, rem in zip(active, remaining_tokens):
if rem <= 1:
req.completion_time = current_time_ms
completed_requests.append(req)
stats.total_tokens_generated += req.max_output_tokens
latency = req.completion_time - req.arrival_time * 1000
stats.latencies_ms.append(latency)
else:
new_active.append(req)
new_remaining.append(rem - 1)
active = new_active
remaining_tokens = new_remaining
else:
# Static batching: process entire batch to completion (max output tokens)
max_steps = max(remaining_tokens)
current_time_ms += max_steps * ms_per_decode_step
for req, rem in zip(active, remaining_tokens):
req.completion_time = current_time_ms
completed_requests.append(req)
stats.total_tokens_generated += req.max_output_tokens
latency = req.completion_time - req.arrival_time * 1000
stats.latencies_ms.append(latency)
active = []
remaining_tokens = []
stats.total_requests = len(completed_requests)
stats.total_time_seconds = current_time_ms / 1000.0
return statsBenchmarking Batch Size Impact
Now we measure how batch size affects throughput and latency across both batching strategies. This is the kind of sweep you would run before deploying a model to production.
batch_sizes_to_test = [1, 2, 4, 8, 16, 32, 64]
continuous_results = []
static_results = []
small_workload = generate_workload(n_requests=200, rng_seed=0)
for bs in batch_sizes_to_test:
cont_stats = simulate_serving(
small_workload, batch_size=bs, use_continuous_batching=True
)
stat_stats = simulate_serving(
small_workload, batch_size=bs, use_continuous_batching=False
)
continuous_results.append(cont_stats)
static_results.append(stat_stats) Batch Cont Throughput Static Throughput Cont P99 (ms) Static P99 (ms)
--------------------------------------------------------------------------------
1 192.1 t/s 192.1 t/s 150766 ms 150766 ms
2 369.4 t/s 258.6 t/s 68961 ms 107377 ms
4 672.8 t/s 384.9 t/s 28528 ms 66447 ms
8 1139.3 t/s 552.7 t/s 9205 ms 40528 ms
16 1749.7 t/s 879.7 t/s 2778 ms 18298 ms
32 2419.4 t/s 1356.7 t/s 4182 ms 6174 ms
64 2954.5 t/s 1937.4 t/s 6637 ms 4645 msThe results show token throughput rising substantially with batch size for both strategies, with continuous batching consistently delivering higher throughput because it keeps batch slots occupied throughout generation. At small batch sizes, the two strategies perform similarly because there is less opportunity for idle slots in a small batch. At large batch sizes, the divergence grows because longer requests keep more slots idle under static batching.
The dual-panel view in @fig-batch-size-throughput-latency makes the tradeoff concrete. At batch size 1, the simulated server cannot keep up with the fixed offered load, so requests accumulate in the queue and P99 latency becomes extreme. Increasing the batch size raises throughput sharply and drains that backlog, reducing latency until the system reaches a low-latency knee. Beyond the knee, throughput gains flatten and queueing pressure begins to reappear. The useful operating point is therefore the smallest batch size that clears the offered load with enough throughput headroom.
Notice also that continuous batching reaches the low-latency region at a smaller batch size than static batching while delivering higher throughput throughout the sweep. This is because continuous batching eliminates the "wait for the slowest request" effect that causes static batching's occasional very long latencies. When the slowest request in a static batch runs to 500 tokens, every other request in that batch must wait those 500 steps. In continuous batching, short requests finish and leave the batch long before the long request completes. At the largest tested batches, both strategies have enough capacity to clear the offered load, so their P99 latencies converge.
A Worked Throughput Calculation
To make the concepts concrete, let us trace through a specific scenario. Suppose you are serving a 7B model on an A100 80GB GPU. Your target is to serve 50 concurrent users at a P99 latency under 3 seconds, and you expect an average output of 150 tokens at 5 ms per decode step.
Step 1: Calculate the expected time to generate a 150-token response at batch size .
At batch size 32, this time is shared across 32 concurrent requests. The GPU runs all 32 simultaneously, so the wall clock time is still roughly 750 ms (plus queuing if the batch is full).
Step 2: Calculate token throughput at batch size 32.
Step 3: Estimate memory usage. Model weights at 14 GB, KV cache at 1 MB per token. For 32 requests at 150 tokens average output plus 256 tokens average prompt, the working KV cache is roughly . Total: 27 GB, well within the 80 GB limit.
Step 4: Check if the batch size matches the roofline. With the crossover around batch size 156 for this model, batch size 32 is well into the memory-bound regime. You are not saturating GPU compute. This tells you that if you can increase batch size to 64-128, you will get proportionally higher throughput with minimal additional latency cost (the decode step time barely changes in the memory-bound regime as batch size increases).
Step 5: Verify latency SLO. At batch size 32 with continuous batching, P99 latency is dominated by the long-tail output lengths. The P99 output length for a log-normal distribution with mean 150 is typically around 400-600 tokens. At 5 ms per step, P99 latency is roughly , right at your 3-second SLO.
This calculation suggests that batch size 32 barely meets the SLO. Going to batch size 64 would improve throughput but might push P99 latency over 3 seconds due to longer queuing. The right answer is to run the simulation and profile at sustained load, using the batch size that keeps P99 under 3 seconds with a 10-20% margin.
Key Parameters
The key parameters for throughput optimization are:
- max_batch_size: The maximum number of requests processed simultaneously. Controls the peak throughput and memory budget. Set based on available GPU memory minus model weight overhead.
- max_wait_time_ms: Maximum time a request waits before being dispatched even with an incomplete batch. Lower values reduce latency at the cost of smaller, less efficient batches.
- continuous_batching: Whether to use iteration-level scheduling (recommended for production). Most modern serving frameworks (vLLM, TGI, TensorRT-LLM) enable this by default.
- tensor_parallel_degree: Number of GPUs to split model layers across within a node. Increases memory capacity and throughput at the cost of all-reduce communication overhead.
- pipeline_parallel_degree: Number of nodes to pipeline across. Enables very large models but introduces bubble overhead that requires large micro-batch sizes to amortize.