Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Continuous Batching: Optimizing LLM Inference Throughput › Real-World Performance

Michael BrenndoerferJanuary 18, 20263 min read

Language AI HandbookAll chapters

Part 42 of 67Inference Optimization

Chapter 325Continuous Batching: Optimizing LLM Inference Throughput › Real-World Performance

Reading section 11 of 14 · Real-World Performance · 3 minSection 11 of 14 · 3 min

Section 11 of 14 · 3 min read

Continuous Batching: Optimizing LLM Inference Throughput › Real-World PerformanceOne section · Sections in this chapter11 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Production continuous batching systems like vLLM, TensorRT-LLM, and Text Generation Inference report significant improvements over static batching. These systems combine continuous batching with paged attention, optimized kernels, and advanced scheduling to reach impressive throughput on commodity hardware.

The gains in production are often even larger than the simulations suggest, because real workloads have extreme variance. A production LLM serving endpoint might receive requests ranging from two-token completions to 4000-token document generations, all mixed together in the same serving pool. Static batching degrades catastrophically in this setting; continuous batching handles it naturally.

In[35]:

Code

# Simulated benchmark data based on published results
benchmark_data = {
    "batch_size": [1, 4, 8, 16, 32],
    "static_throughput": [15, 45, 70, 95, 110],  # tokens/sec
    "continuous_throughput": [15, 58, 120, 210, 350],  # tokens/sec
    "static_latency_p50": [65, 85, 110, 150, 220],  # ms
    "continuous_latency_p50": [65, 70, 75, 85, 100],  # ms
}

Out[36]:

Visualization

Line plot showing throughput versus batch size for static and continuous batching. Continuous batching grows steeply while static flattens. — Throughput scaling with maximum batch size. Continuous batching scales nearly linearly with batch size, reaching 350 tokens per second at batch size 32, while static batching plateaus around 110 because synchronization overhead from waiting for batch-mates dominates at larger batches.

Line plot showing P50 latency versus batch size for static and continuous batching. Static latency rises steeply while continuous stays nearly flat. — P50 latency scaling with maximum batch size. Continuous batching maintains stable latency (65 to 100 ms) even as batch size grows, while static batching latency more than triples (65 to 220 ms) because larger batches mean longer waits for the slowest concurrent request.

The throughput advantage of continuous batching compounds at larger batch sizes because GPU utilization stays high even as requests complete at different times. Static batching's throughput plateaus as the "waiting for slowest" overhead dominates, while continuous batching continues to scale. The latency curves tell an equally important story: continuous batching's latency grows slowly because each request's wait time is bounded by how long it takes to process its own tokens, not the tokens of batch-mates.