Throughput Optimization › Throughput Optimization

When you deploy a language model in production, latency tells only half the story. A system that responds in 100 milliseconds to one user may completely collapse when 100 users arrive simultaneously. The dimension you also need to master is throughput: how many requests your system can process per unit of time. Latency is personal. Throughput is systemic.

Throughput optimization is the art of squeezing maximum utilization from your GPU hardware. An idle GPU costs the same as a busy one, so every second of underutilization is money left on the table. More importantly, throughput determines whether your infrastructure can serve your user base at all. A model that handles 10 requests per second cannot support a product with 1,000 concurrent users without massive over-provisioning, and over-provisioning at GPU prices quickly becomes the largest engineering cost on the balance sheet.

What makes LLM throughput uniquely challenging is the nature of autoregressive generation. Unlike image classifiers or embedding models, where a single forward pass produces the final result, language models generate tokens one at a time. Each token generation step reads the entire model parameter set from GPU memory, then writes one token into the KV cache. This sequential dependency creates bottlenecks that do not appear in other deep learning workloads, and it means that techniques borrowed from classical deep learning inference frequently do not apply without significant adaptation.

This chapter covers the full throughput optimization stack. You will learn how to measure throughput correctly, understand why naive batching fails for language models, study how continuous batching solves the problem, examine the parallelism strategies used at multi-GPU scale, and reason about the fundamental tradeoffs between latency and throughput. By the end, you will have both the conceptual framework and the practical tools to reason about and improve throughput in real deployments.