Continuous Batching: Optimizing LLM Inference Throughput › Worked Example: Step-by-Step Trace

To cement the intuition, let's trace through a small concrete example comparing both strategies on the same five requests. This walkthrough shows exactly what happens at each iteration and why the results differ.

Suppose we have a maximum batch size of 3, and five requests arrive all at once with the following output lengths: Request A needs 4 tokens, Request B needs 2 tokens, Request C needs 6 tokens, Request D needs 3 tokens, and Request E needs 5 tokens.

Static batching: The system collects all five requests but the batch size is 3, so it forms two batches. Batch 1 contains Requests A, B, C. Batch 2 contains Requests D and E (padded to a batch of 2 since no more requests exist).

For Batch 1, the longest request is C with 6 tokens. All three requests run for 6 iterations:

• Iterations 1-2: A, B, C all active (B finishes at iteration 2 but stays in batch)
• Iterations 3-4: A, B (idle), C all running (A finishes at iteration 4 but stays in batch)
• Iterations 5-6: A (idle), B (idle), C all running

Total slot-iterations for Batch 1: $3 \times 6 = 18$. Useful tokens: $4 + 2 + 6 = 12$. Efficiency: $12/18 \approx 67\%$.

Batch 2 begins only after Batch 1 finishes. The longest of D (3 tokens) and E (5 tokens) is 5, so both run for 5 iterations.

Total slot-iterations for Batch 2: $2 \times 5 = 10$. Useful tokens: $3 + 5 = 8$. Efficiency: $8/10 = 80\%$.

Grand total: 28 slot-iterations, 20 useful tokens, overall efficiency $20/28 \approx 71\%$. Total wall-clock iterations: $6 + 5 = 11$.

Continuous batching: The system loads Requests A, B, C into the three available slots and begins immediately. Requests D and E wait in the queue.

• Iteration 1: A generates token 1, B generates token 1, C generates token 1. No completions. Queue: D, E.
• Iteration 2: A generates token 2, B generates token 2 (B is now complete), C generates token 2. B exits. D enters. Slot composition: A, C, D.
• Iteration 3: A generates token 3, C generates token 3, D generates token 1. No completions. Queue: E.
• Iteration 4: A generates token 4 (A complete), C generates token 4, D generates token 2. A exits. E enters. Slot composition: C, D, E.
• Iteration 5: C generates token 5, D generates token 3 (D complete), E generates token 1. D exits. Queue empty. Slot composition: C, E.
• Iteration 6: C generates token 6 (C complete), E generates token 2. C exits. Slot composition: E.
• Iteration 7: E generates token 3. No completions. Slot composition: E.
• Iteration 8: E generates token 4. No completions. Slot composition: E.
• Iteration 9: E generates token 5 (E complete). All done.

Total wall-clock iterations: 9. Total slot-iterations used: let's count. Iterations 1-4 had 3 active requests each (12), iteration 5 had 2 (2), iteration 6 had 2 (2), iterations 7-9 had 1 each (3). Total: $12 + 2 + 2 + 3 = 19$ slot-iterations for 20 useful tokens. Efficiency: $20/19$ which rounds to approximately 100% once you account for the single unavoidable idle slot-iteration when only E remained.

Static batching needed 11 wall-clock iterations and 28 slot-iterations. Continuous batching needed 9 wall-clock iterations and used only 19 slot-iterations. This is a modest example; with larger batches and higher variance, the gap grows substantially.