Continuous Batching: Optimizing LLM Inference Throughput › Throughput Analysis

Let's compare the throughput of static and continuous batching quantitatively. Understanding the mathematical relationship between these approaches helps explain why continuous batching delivers such significant improvements and under what conditions those improvements are largest.

The analysis that follows formalizes the intuition we built earlier: static batching allocates computational capacity based on the worst case in each batch, while continuous batching adapts to the actual workload token by token. The efficiency gap between these strategies grows with the variance of output lengths, which is exactly what characterizes real LLM workloads.

To understand the efficiency difference mathematically, we need to formalize what happens in each batching strategy. The key insight is that static batching allocates computational capacity based on the worst case, while continuous batching adapts to the actual workload.

For static batching with a fixed batch size $B$ containing requests with output lengths $L_1, L_2, \ldots, L_B$, the total computational time is determined by the longest request. Every request in the batch must wait for this slowest member, regardless of whether it has long since completed. The total computational capacity allocated, measured in token-slots (one slot processing one token), is:

$$C_{\text{static}} = B \times \max(L_1, L_2, \ldots, L_B)$$

where:

• $C_{\text{static}}$: the total number of processing slots consumed (iterations $\times$ batch size)
• $B$: the fixed batch size
• $L_i$: the output length of the $i$-th request
• $\max(L_1, \ldots, L_B)$: the length of the longest sequence in the batch

This formula captures the basic problem: we allocate capacity based on the maximum, not the sum. Every slot runs for as many iterations as the longest sequence demands, even if that slot has been creating padding for 90% of those iterations.

We can calculate the compute efficiency $\eta_{\text{static}}$ as the ratio of useful work to total capacity. Useful work is the actual tokens generated, while total capacity is the processing slots allocated:

$$\eta_{\text{static}} = \frac{\sum_{i=1}^{B} L_i}{B \times \max_{j} L_j}$$

where:

• $\eta_{\text{static}}$: the computational efficiency (between 0 and 1)
• $\sum_{i=1}^{B} L_i$: the total number of generated tokens (useful work)
• $B \times \max_{j} L_j$: the total capacity allocated during the batch's lifetime

Notice that efficiency depends heavily on the distribution of output lengths. If all requests have the same length, the numerator equals the denominator and efficiency is 100%. But if one request is much longer than the others, efficiency drops sharply.

To make this concrete, consider a batch of 8 requests where 7 need 20 tokens and 1 needs 200 tokens. We can work through the efficiency calculation step by step:

$$\begin{aligned} \sum_{i=1}^{8} L_i &= 7 \times 20 + 200 = 340 \text{ tokens} \\ B \times \max_j L_j &= 8 \times 200 = 1{,}600 \text{ token-slots} \\ \eta_{\text{static}} &= \frac{340}{1{,}600} \approx 0.21 \end{aligned}$$

The system performs 1600 computational operations but only 340 of them produce useful output. That is 79% wasted compute, or a 21% efficiency rating. The single outlier request forces seven other requests to continue burning resources for 180 extra iterations each.

For continuous batching, the situation is fundamentally different. Assuming the request queue remains populated (perfect slot filling), the system avoids waiting for long requests. When a request completes, its slot immediately fills with a new request. The total iterations required approach the theoretical minimum:

$$T_{\text{continuous}} \approx \frac{\sum_{i=1}^{N} L_i}{B}$$

where:

• $T_{\text{continuous}}$: the approximate total iterations needed
• $N$: the total number of requests in the workload
• $L_i$: the output length of the $i$-th request
• $B$: the maximum batch size (concurrency limit)
• $\sum_{i=1}^{N} L_i$: the total number of tokens to be generated across all requests

This formula shows that continuous batching distributes work evenly across the available batch slots. The total iterations is simply the total tokens divided by the parallelism factor. Doubling $B$ halves the time, and adding more requests to a non-empty queue barely changes the total time because they fill slots freed by completions.

We can derive the efficiency $\eta_{\text{continuous}}$ by comparing useful work to allocated capacity:

$$\begin{aligned} \eta_{\text{continuous}} &= \frac{\text{Useful Work}}{\text{Total Capacity}} \\ &= \frac{\sum_{i=1}^{N} L_i}{T_{\text{continuous}} \times B} \\ &\approx \frac{\sum_{i=1}^{N} L_i}{\left( \frac{\sum_{i=1}^{N} L_i}{B} \right) \times B} \\ &= 1 \end{aligned}$$

This shows that efficiency approaches 100% because the total capacity used matches the useful work. Every slot-iteration produces a real output token rather than padding. The approximation becomes exact when the queue is always full and requests are perfectly divisible across slots.

The practical implication is that continuous batching eliminates the sensitivity to output length distribution. Whether requests generate 10 tokens or 1000, the throughput stays near the theoretical maximum. Static batching is at the mercy of whoever happens to be in the same batch as you, while continuous batching processes everyone at their natural speed.

Let's run a more complete simulation comparing the two approaches across a realistic workload. Theory tells us continuous batching should win, but simulation reveals the magnitude of improvement and how it varies with workload characteristics.