Throughput Optimization › Summary

Throughput optimization for language models requires understanding several interlocking concepts:

• Measuring throughput correctly means distinguishing request throughput from token throughput, measuring under sustained load rather than brief bursts, reporting percentile latencies alongside throughput, and understanding the throughput-latency curve that governs all optimization decisions.
• Static batching is simple but wasteful: early-finishing requests hold batch slots idle, and variable-length inputs require padding waste. It is a poor fit for the length-variable outputs common in production, where the gap between the shortest and longest request can span two orders of magnitude.
• Continuous batching solves these problems by reassigning batch slots at every generation step. It provides 2-4x throughput improvement over static batching on mixed-length workloads and is the standard approach in modern serving frameworks. PagedAttention provides the memory management foundation that makes efficient continuous batching possible.
• Batch size tuning involves finding the point where the GPU transitions from memory-bandwidth-bound to compute-bound. The crossover batch size for a 7B model on an A100 is roughly 150. Below this threshold, larger batches dramatically improve throughput by amortizing weight-read costs. Above it, the gains diminish.
• Parallelism strategies scale beyond a single GPU. Data parallelism provides linear scaling with no communication overhead but requires the full model on each GPU. Tensor parallelism splits layers across GPUs with all-reduce overhead, allowing models that exceed single-GPU memory. Pipeline parallelism assigns layers to GPU stages with bubble overhead, scaling across nodes. Production deployments combine all three based on model size and hardware topology.
• Tradeoffs are unavoidable. Throughput optimization competes with latency (via batch queueing), output quality (via quantization), reliability (via parallelism failure domains), and operational simplicity (via system complexity). SLO constraints determine how each decision balances these tensions.

The next chapter covers monitoring in production, where you will learn how to observe these throughput metrics continuously and detect when serving performance degrades before it affects users.