Throughput Optimization: Batch Size, GPU Utilization

When you deploy a language model in production, latency tells only half the story. A system that responds in 100 milliseconds to one user may completely collapse when 100 users arrive simultaneously. The dimension you also need to master is throughput: how many requests your system can process per unit of time. Latency is personal. Throughput is systemic.

Throughput optimization is the art of squeezing maximum utilization from your GPU hardware. An idle GPU costs the same as a busy one, so every second of underutilization is money left on the table. More importantly, throughput determines whether your infrastructure can serve your user base at all. A model that handles 10 requests per second cannot support a product with 1,000 concurrent users without massive over-provisioning, and over-provisioning at GPU prices quickly becomes the largest engineering cost on the balance sheet.

What makes LLM throughput uniquely challenging is the nature of autoregressive generation. Unlike image classifiers or embedding models, where a single forward pass produces the final result, language models generate tokens one at a time. Each token generation step reads the entire model parameter set from GPU memory, then writes one token into the KV cache. This sequential dependency creates bottlenecks that do not appear in other deep learning workloads, and it means that techniques borrowed from classical deep learning inference frequently do not apply without significant adaptation.

This chapter covers the full throughput optimization stack. You will learn how to measure throughput correctly, understand why naive batching fails for language models, study how continuous batching solves the problem, examine the parallelism strategies used at multi-GPU scale, and reason about the fundamental tradeoffs between latency and throughput. By the end, you will have both the conceptual framework and the practical tools to reason about and improve throughput in real deployments.

Before you optimize throughput, you need to measure it correctly. This sounds simple but involves several subtleties that can lead to misleading benchmarks if you are not careful. A benchmark that overstates throughput by 3x will cause you to under-provision hardware and produce an outage the first time real users arrive.

Language model throughput can be measured at two granularities:

• Request throughput: the number of complete request-response cycles per second. This is the most relevant metric for capacity planning, since each request represents a user interaction.
• Token throughput: the total number of tokens generated per second across all concurrent requests. This is more useful for understanding hardware utilization, since the GPU generates tokens regardless of how they group into requests.

The two metrics are related. If your average response contains $L$ tokens and your system processes $R$ requests per second, then token throughput is approximately $R \times L$. However, token throughput depends on output length, which varies by request. A system serving short answers shows different token throughput than one generating long documents, even at identical GPU utilization.

For capacity planning, always report both. Token throughput tells you how close you are to the hardware ceiling. Request throughput tells you how many users you can serve. A system running at 10,000 tokens/second with an average output of 100 tokens handles 100 requests per second. The same system with an average output of 500 tokens handles only 20 requests per second, even though GPU utilization is identical.

Time-to-first-token (TTFT) and inter-token latency (ITL) are also worth tracking alongside throughput. TTFT measures how quickly the model starts generating after receiving a request, which depends on prefill compute time and queueing delays. ITL measures the gap between successive generated tokens, which reflects the decode step time. Users perceive both: a long TTFT feels like the system is hanging even if generation is fast once it starts, while high ITL makes streaming responses feel choppy.

A subtle but important distinction: the throughput you measure in isolation differs from the throughput you achieve under sustained load.

Peak throughput is the maximum rate when the system is fully saturated: every GPU thread is busy, batches are full, and no resources are idle. Benchmarks that ramp up load until the system saturates measure this.

Sustained throughput is the rate you can maintain over minutes or hours without degradation. It is limited by memory accumulation (KV cache growth), garbage collection overhead in the serving framework, and thermal throttling on some hardware. On many systems, sustained throughput runs 10-20% below peak because of these ongoing overheads.

Burst throughput is the rate achievable for short windows when internal queues are empty and memory is fresh. It may exceed sustained throughput because the system has not yet accumulated overhead. Burst throughput is what many public benchmarks report, making them optimistic for real production planning.

Production planning should use sustained throughput, measured over at least five minutes of continuous load, not brief benchmark windows. The difference between burst and sustained throughput is the difference between a benchmark and reality.

Throughput and latency are not independent: they form a curve. At low request rates, the system is underloaded. Latency is near minimum, dominated by a single request's computation, but throughput is low because the GPU is idle much of the time. The CPU is submitting work faster than it accumulates, so the GPU sits waiting.

As request rate increases, the system fills its batch capacity. Latency increases because requests queue for batch slots, but throughput rises because the GPU is now busier. Eventually the system saturates: latency climbs steeply while throughput plateaus. Beyond saturation, adding more requests only causes queuing without improving throughput. The queue grows without bound and latency approaches infinity in the mathematical model, or your queue drops requests in the practical implementation.

The optimal operating point for most production systems is just below saturation: throughput is near-maximum, and latency is still acceptable. This is not a fixed point on the curve: it moves based on your latency SLO. A system with a 2-second SLO can run at higher utilization than one with a 100-millisecond SLO, because the latency budget absorbs more queuing delay.

Understanding the shape of this curve for your specific model and hardware is essential before setting capacity targets. Systems with highly variable output lengths (like chat models) have broader curves because continuous batching smooths the saturation behavior. Systems with fixed output lengths (like code completion at a fixed number of tokens) have sharper saturation transitions.

The curve in @fig-throughput-latency-curve illustrates the fundamental tension. Notice how throughput rises nearly linearly until saturation and then plateaus, while latency stays low until the system is about 80% loaded and then climbs steeply. The green shading marks the practical operating zone: you want to keep the system loaded enough to maximize GPU utilization but not so loaded that latency blows up.

What the curve does not show is how it shifts under different conditions. Adding GPUs shifts the saturation point rightward without changing the curve shape. Enabling continuous batching shifts the saturation point rightward and also changes the steepness of the latency rise, because batch slots refill continuously rather than discretely. Understanding both effects helps you reason about the return on investment of each optimization.

A common mistake in throughput benchmarking is reporting average latency alongside throughput. For production capacity planning, percentile latencies matter far more. If your system serves 1,000 requests per second with a mean latency of 200 ms but a P99 latency of 4 seconds, then 10 users per second experience a 4-second wait. In a production API, P99 latency frequently triggers SLO violations and user complaints.

The reason percentile latencies diverge from averages under load is the presence of long requests in the batch. A batch containing a 500-token request and 15 short requests has the short requests waiting for those 500 decode steps. Under static batching, this causes occasional enormous latency spikes at P99. Continuous batching reduces but does not eliminate this effect, because long requests still hold batch slots.

Always benchmark at multiple load levels, report P50, P95, and P99 latencies, and hold the system at each load level for at least five minutes before recording. Load spikes that last only a few seconds often look fine in benchmarks but cause cascading timeouts in production.

The most intuitive approach to improving throughput is batching: group multiple requests together and process them as a unit. A single matrix multiplication over a batch of 32 input sequences is much faster than 32 separate matrix multiplications, because modern GPUs excel at wide, parallel operations. This observation is correct as far as it goes, but language model generation introduces complications that make naive batching surprisingly inefficient.

In static batching, you accumulate requests until you have a full batch (or a timeout expires), then process all of them together. Every request in the batch enters the model at the same time and exits at the same time. The batch is treated as an atomic unit from the moment it is assembled to the moment the last token is generated.

The batch size $B$ directly controls throughput. Each forward pass through the model costs roughly $C$ milliseconds of GPU time. A batch of $B$ requests costs approximately $C$ milliseconds and produces $B$ responses, so throughput scales as:

where:

• $B$: the batch size (number of requests processed together)
• $C$: the time in milliseconds for one forward pass over the batch

Doubling $B$ doubles throughput as long as $C$ stays constant, which is approximately true when the model is compute-bound and GPU memory can fit the larger batch. This observation motivated the development of batching in classical deep learning, and it holds reasonably well for single-pass inference tasks like image classification.

For language model generation, however, the formula is deceptive. The forward pass is not a single operation: it is $T$ sequential decode steps, where $T$ is the output length. And the output length is not fixed across requests. This is where static batching breaks down.

Language model generation introduces a critical complication: requests finish at different times. One user asks for a one-sentence summary (20 tokens), another asks for a detailed essay (500 tokens). In static batching, all requests in the batch must finish before any response is returned.

Consider the GPU computation from step 21 onwards when the short request is done. The GPU is generating tokens 21 through 500 for the long request. The short request's batch slot is occupied but producing nothing useful. No new request can enter the batch to fill that slot. The GPU is running decode steps, but only one of the 32 batch slots is doing productive work.

The idle slot problem compounds with batch size. In a batch of 32 requests with a bimodal length distribution (half short at 20 tokens, half long at 500 tokens), after step 20 you have 16 occupied slots and 16 idle slots. You are running at half efficiency for 480 steps. The waste grows with the ratio between the longest and shortest requests.

Worse, the memory occupied by the completed request's KV cache cannot be freed until the batch finishes, limiting how much memory is available for new requests. The KV cache for those 16 finished requests is still allocated, preventing the system from admitting new work even though the slots are logically idle.

Variable-length inputs create a second problem. To batch sequences together, you must pad shorter sequences to match the longest sequence in the batch. If your batch contains one 500-token request and 31 one-sentence requests, those 31 short requests are padded to 500 tokens. The attention mechanism processes all 500 positions for every request, even the empty padding positions.

This waste is not merely theoretical. Consider the attention computation for one layer: it processes a sequence length of 500 for every request. For the 31 padded requests, positions 20-500 are padding and their contribution to the output is zeroed out by the attention mask. The GPU does the arithmetic anyway: loading weights, computing queries, keys, and values, running softmax, applying the mask. All of that computation is discarded.

Depending on the length distribution of your workload, padding waste can reduce effective GPU utilization by 30-70%. You are doing real computation on fake data. For services with highly variable request lengths, which is nearly every production chat or completion service, static batching's padding overhead is severe enough to eliminate most of the throughput benefit of batching at all.

There are partial mitigations: bucketing requests by length and forming batches within length buckets, or using attention implementations that skip padding positions. But these are workarounds for a fundamental architectural mismatch. The real solution is continuous batching.

Continuous batching (also called iteration-level scheduling or in-flight batching) solves the static batching inefficiency by decoupling batch construction from request lifecycle. Instead of assembling a batch and running it to completion, the system re-evaluates the batch composition after every single generation step.

After each decoding iteration, the scheduler checks two things:

1. Which requests in the current batch have just finished (generated an end-of-sequence token or hit the maximum length)?
2. Which new requests are waiting in the queue?

Finished requests are immediately evicted. Their KV cache memory is freed. New requests are inserted to fill the freed slots. The next iteration processes a fresh batch that may contain requests at completely different generation stages.

This seemingly simple change greatly improves utilization. Consider a batch of 8 requests. With static batching, if 6 requests finish at step 20 but 2 continue to step 200, those 6 slots sit idle for 180 steps. With continuous batching, those 6 slots are filled with new requests at step 21. The GPU is doing useful work for those 180 steps instead of waiting.

The scheduler also handles prefill integration naturally. When a new request is admitted to the batch, its prompt tokens are processed in a prefill pass before the first decode step. Modern continuous batching implementations (such as vLLM and TGI) interleave prefill and decode at the step level, so a newly admitted request contributes its first generated token to the output stream within one scheduling interval.

One question that arises when learning about continuous batching is how requests at different generation stages can run together. Request A is at token 150, request B is at token 5, request C was just admitted. How does the GPU handle this?

The answer is that the GPU does not care. Each decode step processes one token per request in the batch, using that request's current KV cache state. The model parameters are shared, and the input to each request's decode step is simply its most recently generated token plus its KV cache. Whether that KV cache has 150 entries or 5 entries is irrelevant to the GPU; it reads the relevant portion and proceeds.

The KV cache for each request grows by one entry per decode step. The memory manager needs to track each request's current KV cache size and ensure new pages are allocated as requests grow. This is where PagedAttention becomes critical, as we discuss below.

The throughput improvement from continuous batching depends on the variance in output lengths across your workload. For uniform output lengths, there is no benefit because all requests finish simultaneously anyway. For highly variable output lengths (typical in production, where users ask a mix of simple and complex questions), the improvement is substantial.

Empirical results from the vLLM evaluation (Kwon et al., 2023) showed 2-4x higher throughput compared to static batching on standard benchmarks, with the improvement growing as batch sizes increased and output length variance widened. Real production systems often see even larger improvements because production length distributions are more variable than benchmark distributions, which tend to use fixed or narrow length ranges.

The intuition for the improvement is simple: every batch slot that static batching leaves idle is a slot that continuous batching fills with real work. The more long requests in your workload, the more idle slots static batching creates, and the more benefit continuous batching delivers.

Continuous batching requires fine-grained memory management. Each active request holds a KV cache: the stored key and value tensors from all previous attention layers and all previous tokens. A 7B-parameter model with 32 attention layers and 32 key/value heads, processing sequences up to 2048 tokens, needs roughly:

where:

• $2$: factor for storing both keys and values
• $L$: number of transformer layers
• $n_{h}$: number of attention heads (or key-value heads in grouped-query attention)
• $d_{h}$: head dimension ($d_{\text{model}} / n_h$)
• $T$: current sequence length
• $\text{bytes\_per\_element}$: typically 2 bytes for FP16

For a 7B model this works out to roughly 1 MB per token of KV cache. A batch of 100 requests at 500 tokens each occupies about 50 GB of KV cache alone, exceeding the memory of a single A100 GPU.

The traditional approach to KV cache allocation pre-allocates a contiguous block of memory for each request, sized for the maximum possible output length. This wastes enormous amounts of memory when requests complete early, because the pre-allocated but unused portions of the block sit empty until the request finishes and the entire block is freed.

PagedAttention (Kwon et al., 2023) addresses this by managing KV cache in fixed-size pages, similar to virtual memory in operating systems. Rather than allocating a contiguous block for each request's full context, the system allocates pages on demand as the sequence grows and reclaims them when the request completes. This eliminates fragmentation and enables the fine-grained memory sharing needed for efficient continuous batching.

The page size is a tunable parameter. Smaller pages reduce internal fragmentation (wasted space within a partially filled page) but increase bookkeeping overhead. Larger pages reduce bookkeeping but increase fragmentation. Most implementations use page sizes between 16 and 64 tokens, which balances these concerns across typical workloads.

PagedAttention also enables memory sharing between requests with the same prompt prefix. In a system serving many users with the same system prompt, the KV cache pages for that shared prefix can be stored once and pointed to by all active requests. This prefix caching can dramatically reduce KV cache memory usage in applications with long, fixed system prompts.

The simulation shows continuous batching requires significantly fewer total GPU steps to process the same set of requests. This translates directly to higher throughput: the GPU spends more of its time generating tokens that users are waiting for, and less time running steps for requests that have already finished.

The magnitude of the improvement depends heavily on the length distribution. A workload with very uniform output lengths shows little improvement. A workload with extreme variance (some requests finishing in 5 tokens, others running to 2,000) shows improvements approaching the ratio of longest to shortest output. This is exactly the distribution you see in production chat services, where users sometimes ask trivial yes/no questions and sometimes request multi-page documents.

The scheduler in a continuous batching system is responsible for several decisions that directly affect throughput:

Admission control decides which waiting requests enter the active batch. A simple scheduler admits requests in arrival order. More sophisticated schedulers use priority queues, preempt long-running requests when high-priority work arrives, or reserve capacity for interactive users versus batch users.

Memory management tracks available KV cache pages and denies admission when memory is insufficient. Without this gate, requests that exceed available memory will cause out-of-memory errors that crash the serving process. The scheduler must model expected KV cache growth to prevent over-admission.

Preemption handles the case where a long-running request occupies a slot that a high-priority request needs. The scheduler can checkpoint the long request's KV cache state, evict it from the batch, process the high-priority request, then restore the long request when capacity is available. This adds latency to the preempted request but protects latency for high-priority work. Most production schedulers implement preemption as a last resort, triggering only when memory pressure is severe.

Choosing the right batch size is an empirical exercise, but several principles guide the search. You cannot simply pick the largest batch that fits in memory: you need to understand the relationship between batch size, GPU compute utilization, and the memory-bandwidth bottleneck that governs autoregressive decoding.

The first constraint is GPU memory. For each request in a batch, you must hold:

• The model weights (fixed cost, shared across all requests)
• The activation tensors during the forward pass (proportional to batch size and sequence length)
• The KV cache for all previous tokens (proportional to batch size and sequence length)

As batch size grows, the KV cache and activations grow proportionally. At some point, you exceed available GPU memory and must either use a smaller batch or switch to a smaller model.

The approximate memory budget for a batch is:

where:

• $M_{\text{weights}}$: fixed memory for model parameters (roughly $2 \times \text{params}$ bytes in FP16)
• $B$: batch size
• $M_{\text{activations}}$: activation memory per request (proportional to sequence length)
• $M_{\text{kv\_cache}}$: KV cache memory per request (proportional to context length and layer count)

For a 7B model on an A100 80GB GPU, model weights consume about 14 GB, leaving roughly 60 GB for batch data. If each request uses 600 MB at a 2048-token context, maximum batch size is about 100 requests. In practice, you leave a safety margin of 10-15% to avoid out-of-memory errors from measurement imprecision and batch composition variance.

This calculation clarifies why KV cache quantization (computing and storing KV entries in INT8 or FP8 rather than FP16) has such a large impact. Halving KV cache memory roughly doubles the batch size you can support at a given context length, which doubles throughput for memory-bound workloads.

GPUs have two bottlenecks: compute (how fast arithmetic operations run) and memory bandwidth (how fast data moves from HBM memory to the compute units). Large language model inference is almost entirely memory-bandwidth-bound during token generation, because the prefill of a single token reads the entire model weight matrix once but performs only a small number of arithmetic operations per byte of weight data.

To understand why, consider a single linear layer with weight matrix $W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$ during decode. For a batch of $B$ tokens, the operation is $y = xW$ where $x \in \mathbb{R}^{B \times d_{\text{in}}}$. The number of floating-point operations is $2 \times B \times d_{\text{in}} \times d_{\text{out}}$. The number of bytes read from HBM is approximately $d_{\text{in}} \times d_{\text{out}} \times 2$ bytes (for FP16 weights, loaded once). The ratio of operations to memory access is:

The arithmetic intensity equals the batch size. At $B = 1$, you perform 1 floating-point operation per byte read. At $B = 100$, you perform 100 operations per byte read. The A100 GPU has a peak arithmetic intensity (the "roofline") of roughly 156 FLOP/byte at its memory bandwidth and FP16 peak throughput. You need a batch size of about 156 just to reach the boundary between memory-bound and compute-bound operation.

Increasing batch size helps by amortizing the weight-read cost over more requests. With batch size $B$, each weight-read generates $B$ tokens instead of one. The effective memory efficiency is:

At small $B$, the denominator is dominated by $\text{param\_bytes}$ (reading the weights once), so arithmetic intensity is low and the GPU is memory-bound. As $B$ grows, the numerator grows faster, and eventually the GPU becomes compute-bound. This crossover point is the sweet spot for batch size: you want to be just above it, where adding more to the batch no longer helps because you have already saturated the compute units.

The crossover point shows the minimum batch size needed to saturate GPU compute. Below this, you are underutilizing the GPU because the hardware spends most of its time waiting for memory reads. Above this, you are in the compute-bound regime and additional batch size yields diminishing throughput gains.

This analysis has practical implications beyond just picking a batch size. It tells you that:

• Small batch sizes (1-8) are particularly wasteful because the weight memory reads dominate. Moving from batch size 1 to 8 improves throughput almost eightfold with minimal added latency.
• The return on increasing batch size flattens significantly beyond the crossover point. Going from batch size 200 to 400 improves throughput much less than going from 1 to 2.
• Quantization (reducing bytes per weight) shifts the roofline intensity downward, meaning you reach the compute-bound regime at a smaller batch size. INT8 quantization roughly halves the bytes per weight, halving the crossover batch size and making it easier to saturate GPU compute.

The analysis above applies specifically to the decode phase. During prefill, the arithmetic intensity is much higher because you process a full prompt of $T$ tokens simultaneously. For a 256-token prompt, the effective arithmetic intensity during prefill is approximately $256 \times$ higher than for a single decode step at the same batch size.

This difference explains why prefill and decode have different optimal batch sizes. For prefill, even batch size 1 may be compute-bound on long prompts. For decode, you often need batch sizes in the hundreds to reach compute saturation. This asymmetry motivates prefill-decode disaggregation, which we cover later in the chapter.

In practice, you find the optimal batch size by profiling, not by calculation alone. The memory estimates above involve many approximations, and the actual throughput depends on framework overhead, kernel selection, and driver behavior. A practical search procedure:

1. Start with a conservative batch size (e.g., 8) that definitely fits in memory.
2. Measure throughput (tokens per second) and GPU utilization with nvidia-smi or a profiler.
3. Double the batch size and repeat.
4. Stop when either GPU memory is exhausted or throughput improvement falls below 10%.
5. Back off slightly from the maximum to leave headroom for request length variance.

This procedure takes 30-60 minutes for a typical model and hardware combination, and the results are invaluable for production capacity planning. You will often discover that your initial batch size was 5-10x below optimal.

When a single GPU cannot hold the model or cannot achieve sufficient throughput, you must distribute the work across multiple GPUs. Several parallelism strategies exist, each with distinct tradeoffs. Choosing the right strategy, or the right combination, depends on model size, hardware topology, and whether you are optimizing for latency or throughput.

Tensor parallelism splits individual weight matrices across multiple GPUs. For a linear layer $y = Wx$, you can split $W$ along either the row or column dimension and assign each partition to a different GPU. After the split computation, an all-reduce operation aggregates results.

To be concrete: a feed-forward layer in a transformer with hidden dimension 8192 has weight matrices of shape $[8192, 32768]$ (for the expansion) and $[32768, 8192]$ (for the projection). With tensor parallelism degree $T = 4$, each GPU holds $[8192, 8192]$ of the expansion matrix. All four GPUs receive the same input $x$ (broadcast), compute their partial output in parallel, then sum the results via all-reduce.

The critical property of tensor parallelism is that it operates within a single layer. Every token in a batch passes through every GPU at every layer, requiring a synchronization (all-reduce) after each attention and feed-forward layer. For a model with $N$ layers, you perform $2N$ all-reduce operations per forward pass.

The communication cost of all-reduce scales with model dimension and the number of GPUs. On NVLink-connected GPUs (within a single node), all-reduce bandwidth is roughly 600 GB/s, making tensor parallelism efficient up to about 8 GPUs per node. Across nodes (using InfiniBand), all-reduce is 10-30x slower and tensor parallelism becomes communication-bound.

Tensor parallelism reduces peak memory per GPU by the parallelism degree $T$ and provides roughly $T$x throughput improvement for compute-bound operations, partially offset by communication overhead. The memory reduction is multiplicative with model size: a 70B model that does not fit on one 80GB A100 can fit on 4 A100s with $T = 4$. This is the primary use case for tensor parallelism on large models.

Pipeline parallelism assigns different layers to different GPUs. GPU 0 runs layers 1-8, GPU 1 runs layers 9-16, and so on. A micro-batch moves through GPUs sequentially, like an assembly line. Unlike tensor parallelism, the communication between pipeline stages is the activation tensors passed from one stage to the next, which is much smaller than the all-reduce traffic of tensor parallelism.

The key challenge is pipeline bubbles: the idle time that occurs while waiting for the previous GPU to finish its stage. With $K$ pipeline stages and micro-batch size $m$, the fraction of time wasted on bubbles is approximately:

where:

• $K$: the number of pipeline stages (GPUs in the pipeline)
• $m$: the number of micro-batches in flight simultaneously

To minimize bubble time, you need many micro-batches flowing through the pipeline simultaneously. This requires a large overall batch, which may conflict with latency requirements. Pipeline parallelism is best suited for large-batch, throughput-focused serving.

For example, with $K = 4$ pipeline stages and $m = 16$ micro-batches, the bubble fraction is $(4 - 1) / (4 - 1 + 16) = 3 / 19 \approx 16\%$. With $m = 4$, the bubble fraction rises to $3 / 7 \approx 43\%$. This shows why pipeline parallelism needs large batches to be efficient: small batch sizes waste nearly half the GPU time on bubbles.

Pipeline parallelism requires inter-GPU communication only at layer boundaries (to pass activations from one stage to the next). This communication is much lighter than the all-reduce in tensor parallelism, making pipeline parallelism viable across nodes connected by InfiniBand. The activation tensor at a layer boundary has size $B \times d_{\text{model}} \times 2$ bytes, which for a batch of 32 and dimension 4096 is about 500 KB per step, easily handled by 100 Gb/s InfiniBand.

Data parallelism is the simplest strategy: run full copies of the model on multiple GPUs and route different requests to different replicas. There is no communication between replicas during inference. A load balancer distributes incoming requests across replicas, and each replica operates independently.

Data parallelism provides linear throughput scaling: $N$ replicas handle $N$x more requests. Its limitation is that each replica must hold the full model in memory, so it requires GPUs large enough to fit the model. For a 7B model in FP16, each replica needs at least 14 GB plus KV cache, so an A100 80GB can host one replica with plenty of memory for batch data.

Data parallelism is the go-to strategy for medium-sized models (under roughly 10B parameters) on modern 80GB GPUs. It provides maximum throughput per dollar because there is no communication overhead between replicas. It also provides natural fault tolerance: if one GPU fails, the remaining replicas continue serving. This is a significant operational advantage over tensor and pipeline parallelism, where a single GPU failure takes down the entire inference group.

The trade-off is cost: you need enough hardware to fit one model copy per replica, and replicas do not share parameters. For a 70B model requiring 4 GPUs for tensor parallelism, running 4 data-parallel replicas requires 16 GPUs total. Whether this is better than a single 4-GPU tensor-parallel group depends on your throughput requirements and whether the 70B model's quality advantage justifies the cost.

Large models (70B+) require combining strategies. A common pattern is:

• Tensor parallelism within a node: Use NVLink bandwidth for low-latency all-reduce across 4-8 GPUs on one machine.
• Pipeline parallelism across nodes: Divide layers across multiple machines, using InfiniBand to pass activations between stages.
• Data parallelism for replicas: Run multiple instances of the tensor+pipeline-parallel model to handle additional load.

For example, a 70B model might use tensor parallelism degree 4 (4 GPUs per node), pipeline parallelism degree 2 (2 nodes), and data parallelism degree 4 (4 replicas of the 2-node setup), for a total of 32 GPUs. This configuration balances the communication overhead of each strategy against its memory reduction benefits.

The selection process for combining strategies is not purely mathematical; it involves empirical benchmarking. The relative importance of NVLink vs. InfiniBand bandwidth, the specific layer dimensions of the model, and the memory bandwidth of the hardware all interact in ways that are difficult to predict analytically. Most teams run a grid search over (tensor_parallel_degree, pipeline_parallel_degree) configurations and select the one with the best sustained throughput at their target latency.

The visualization reveals the key insight: no single parallelism strategy dominates. Data parallelism has zero communication overhead but requires each GPU to hold the full model. Tensor parallelism reduces memory but introduces frequent all-reduce communication. Pipeline parallelism balances both but introduces bubble time. Real deployments select the combination that fits their hardware topology and model size.

Every throughput optimization involves tradeoffs. Understanding them prevents optimizing one dimension at the cost of another. In production systems, you will regularly face decisions where the highest-throughput configuration is not the right configuration, because it sacrifices latency, quality, or reliability in ways your product cannot accept.

The most fundamental tradeoff is between throughput and latency. To maximize throughput, you want large batches: the GPU is always busy, and each weight-read is amortized over many requests. But large batches mean requests wait in queue for a batch slot, increasing latency.

Several levers control where you sit on this tradeoff:

• Maximum batch size: The hard ceiling on how many requests can be in flight simultaneously. Higher values increase throughput but can spike latency under variable load.
• Maximum queue time: How long a request waits before being processed even with an incomplete batch. Lower values reduce latency but allow smaller, less efficient batches.
• Priority tiers: Some systems maintain separate queues for interactive and batch workloads, allowing different latency targets.

The practical implication is that your SLO sets a ceiling on batch size. If your P99 latency SLO is 500 ms and your model takes 5 ms per decode step, you can afford at most 100 decode steps worth of queueing. At 5 ms per step per batch of 32 requests, this constrains how long requests wait. You then select the largest batch size that keeps you within this constraint.

Different product types have radically different SLO profiles. An interactive chat interface requires P99 latency under 2 seconds to feel responsive. A code completion system in an IDE requires P99 latency under 200 ms to not interrupt the developer's flow. A document summarization pipeline running overnight can tolerate latency of minutes, making throughput optimization essentially unconstrained by latency concerns. Always clarify the SLO before optimizing.

Quantization reduces memory footprint and increases arithmetic intensity, enabling larger batches and higher throughput. But lower-precision arithmetic (INT8, INT4) can degrade output quality, particularly on tasks requiring precise reasoning or mathematical computation.

This tradeoff is workload-dependent. Customer service chatbots may tolerate INT4 quantization with negligible quality loss, while scientific reasoning tasks may show measurable degradation. Always evaluate quality alongside throughput when choosing a quantization scheme. A common framework is to measure quality on a representative held-out set before and after quantization, computing the quality delta relative to the throughput improvement. If INT8 quantization improves throughput by 40% at a quality delta of 0.5%, that is usually an excellent trade. If it improves throughput by 20% at a quality delta of 5%, you need to evaluate whether your use case can absorb that quality loss.

Speculative decoding (covered in the Latency Optimization chapter) provides a different tradeoff: it typically maintains full output quality (because the large model verifies every token) but uses extra compute on speculative drafts. For throughput-focused serving, speculative decoding is usually less beneficial than batching improvements, because it adds overhead per request rather than amortizing costs across requests.

A recent architectural insight is that prefill and decode have fundamentally different compute characteristics and should be handled separately.

Prefill (processing the input prompt) is compute-intensive: you process all input tokens in parallel, which maps well to GPU matrix operations. A long prompt (1,000 tokens) benefits from the full parallelism of the transformer. Arithmetic intensity during prefill is proportional to prompt length, so even single-request prefill saturates the GPU's compute units on long prompts.

Decode (generating output tokens one at a time) is memory-bandwidth-bound: you generate a single token per step, so GPU utilization is low unless you batch many requests together. As we analyzed earlier, you need batch sizes in the hundreds to approach the A100 roofline during decode.

Running both on the same GPU creates contention. Large batches optimize decode (high GPU utilization from many concurrent requests) but create unpredictable latency spikes when a long prefill interrupts decode. When a long prefill arrives on a GPU running a large decode batch, the entire batch must pause or the prefill must wait. Either outcome degrades latency: either the existing decode users experience a pause, or the new user waits unusually long to see their first token.

Disaggregation separates these concerns:

• Prefill servers: Highly capable GPUs (or multi-GPU nodes) that handle prompt processing. These see bursty, compute-heavy workloads and benefit from fast NVLink interconnects that accelerate tensor parallelism.
• Decode servers: Memory-bandwidth-optimized hardware (or commodity GPUs) that handle generation. These benefit from stable, high-concurrency batching and can use more memory-efficient hardware (like older A100s with less compute but adequate bandwidth).

Disaggregation adds operational complexity (you need two fleets with a routing layer between them) but enables better resource matching and eliminates the prefill-decode contention that degrades throughput in mixed workloads. Research systems like DistServe and Splitwise have demonstrated that disaggregation can improve goodput (the fraction of GPU time spent on useful work) by 30-50% compared to co-located serving on the same hardware.

The transfer of KV cache between prefill and decode servers is the main technical challenge. After prefill completes on the prefill server, the KV cache must be transferred to the decode server that will handle generation. This transfer uses the same InfiniBand interconnect as pipeline parallelism communication. For long prompts (thousands of tokens), the KV cache transfer can be substantial, and the routing layer must account for transfer time when making scheduling decisions.

Let us build a practical throughput measurement and optimization toolkit. We will simulate a serving system, measure throughput under different conditions, and visualize the impact of key decisions.

We start by modeling realistic request distributions. Production workloads typically show a mix of short and long requests with a roughly log-normal length distribution. A few very long requests coexist with a large volume of short requests, and the tail is heavier than a normal distribution would suggest.

The workload reflects the log-normal shape typical in production: most requests are short, but a long tail of complex requests generates the majority of total tokens. The P95 output length is substantially higher than the mean, which is precisely what makes continuous batching so valuable. Static batching at this workload would have most batch slots sitting idle while the long requests complete.

Next we build a simple serving simulator that can model both static and continuous batching. The simulator tracks wall-clock simulation time, prefill costs, and per-request completion times.

Now we measure how batch size affects throughput and latency across both batching strategies. This is the kind of sweep you would run before deploying a model to production.

The dual-panel view in @fig-batch-size-throughput-latency makes the tradeoff concrete. At batch size 1, the simulated server cannot keep up with the fixed offered load, so requests accumulate in the queue and P99 latency becomes extreme. Increasing the batch size raises throughput sharply and drains that backlog, reducing latency until the system reaches a low-latency knee. Beyond the knee, throughput gains flatten and queueing pressure begins to reappear. The useful operating point is therefore the smallest batch size that clears the offered load with enough throughput headroom.

Notice also that continuous batching reaches the low-latency region at a smaller batch size than static batching while delivering higher throughput throughout the sweep. This is because continuous batching eliminates the "wait for the slowest request" effect that causes static batching's occasional very long latencies. When the slowest request in a static batch runs to 500 tokens, every other request in that batch must wait those 500 steps. In continuous batching, short requests finish and leave the batch long before the long request completes. At the largest tested batches, both strategies have enough capacity to clear the offered load, so their P99 latencies converge.

To make the concepts concrete, let us trace through a specific scenario. Suppose you are serving a 7B model on an A100 80GB GPU. Your target is to serve 50 concurrent users at a P99 latency under 3 seconds, and you expect an average output of 150 tokens at 5 ms per decode step.

Step 1: Calculate the expected time to generate a 150-token response at batch size $B$.

At batch size 32, this time is shared across 32 concurrent requests. The GPU runs all 32 simultaneously, so the wall clock time is still roughly 750 ms (plus queuing if the batch is full).

Step 2: Calculate token throughput at batch size 32.

Step 3: Estimate memory usage. Model weights at 14 GB, KV cache at 1 MB per token. For 32 requests at 150 tokens average output plus 256 tokens average prompt, the working KV cache is roughly $32 \times 406 \times 1\,\text{MB} \approx 13\,\text{GB}$. Total: 27 GB, well within the 80 GB limit.

Step 4: Check if the batch size matches the roofline. With the crossover around batch size 156 for this model, batch size 32 is well into the memory-bound regime. You are not saturating GPU compute. This tells you that if you can increase batch size to 64-128, you will get proportionally higher throughput with minimal additional latency cost (the decode step time barely changes in the memory-bound regime as batch size increases).

Step 5: Verify latency SLO. At batch size 32 with continuous batching, P99 latency is dominated by the long-tail output lengths. The P99 output length for a log-normal distribution with mean 150 is typically around 400-600 tokens. At 5 ms per step, P99 latency is roughly $600 \times 5 = 3000\,\text{ms}$, right at your 3-second SLO.

This calculation suggests that batch size 32 barely meets the SLO. Going to batch size 64 would improve throughput but might push P99 latency over 3 seconds due to longer queuing. The right answer is to run the simulation and profile at sustained load, using the batch size that keeps P99 under 3 seconds with a 10-20% margin.

The key parameters for throughput optimization are:

• max_batch_size: The maximum number of requests processed simultaneously. Controls the peak throughput and memory budget. Set based on available GPU memory minus model weight overhead.
• max_wait_time_ms: Maximum time a request waits before being dispatched even with an incomplete batch. Lower values reduce latency at the cost of smaller, less efficient batches.
• continuous_batching: Whether to use iteration-level scheduling (recommended for production). Most modern serving frameworks (vLLM, TGI, TensorRT-LLM) enable this by default.
• tensor_parallel_degree: Number of GPUs to split model layers across within a node. Increases memory capacity and throughput at the cost of all-reduce communication overhead.
• pipeline_parallel_degree: Number of nodes to pipeline across. Enables very large models but introduces bubble overhead that requires large micro-batch sizes to amortize.

Throughput optimization is powerful, but it is not free. Understanding the real-world constraints prevents you from running into surprises in production.

The KV cache is the central resource constraint. For long-context models (128K tokens), the KV cache for a single request can exceed the size of the model weights. A 70B model in FP16 uses about 140 GB for weights, but a single 128K-token request in that model uses roughly 64 GB of KV cache. This makes large-batch serving of long-context requests nearly impossible on current hardware without aggressive KV cache compression, quantization, or eviction strategies.

Systems that look excellent in benchmarks with 256-token contexts can fail catastrophically when users submit long documents. A serving system designed for short contexts will hit out-of-memory errors the first time a user submits a 10,000-token document for summarization, even if the model technically supports that context length. You must design your memory management to handle the full range of context lengths that your users will submit, including outliers.

The practical mitigations are:

• KV cache quantization: Store KV entries in INT8 or FP8 instead of FP16, halving memory usage with typically negligible quality loss on most tasks.
• KV cache eviction: When memory is full, evict the KV cache entries for the least recently accessed token positions and recompute them on demand. This trades compute for memory and is expensive for frequently accessed positions.
• Context length limits: Cap the context length at a value your hardware can handle at your target batch size. This is the simplest solution and is appropriate when you know users will not submit very long inputs.
• Streaming prefill: Process very long prompts in chunks, holding only the most recent chunk's activations in memory at once. This reduces peak memory during prefill but does not reduce decode-time KV cache size.

Continuous batching introduces a subtler limitation: head-of-line blocking from very long requests. A request that runs for 2,000 decode steps occupies a batch slot for the entire duration. Other requests that arrive during those 2,000 steps must compete for the remaining batch slots. If several long requests arrive simultaneously and fill the batch, short requests queue behind them even though they would complete in 10 steps.

In practice, you mitigate this by setting a maximum output length policy and by using preemption mechanisms (checkpointing and restarting long requests) in extreme cases. Maximum output length is the simpler solution: truncate generation at some maximum (e.g., 2,048 tokens) and return whatever the model has generated so far. Users who need longer outputs must chunk their requests. Preemption is more complex, requiring the scheduler to checkpoint the long request's KV cache state, free its batch slot, and restore it later. Most production serving frameworks implement maximum output length; preemption is less common because of the operational complexity.

The throughput gains from parallelism strategies come with reliability implications. A tensor-parallel inference group across 8 GPUs fails completely if any one GPU goes down. Data parallelism provides natural fault tolerance (the other replicas continue serving), while tensor and pipeline parallelism create correlated failure domains. In high-availability deployments, you typically run at least two replicas of each parallelism group so that one failure does not take down serving.

The mean time between failures (MTBF) for a single A100 GPU is on the order of 1-3 years, but a cluster of 512 GPUs expects a failure roughly every day. For a deployment using 8-GPU tensor-parallel groups, a single GPU failure takes down 8 GPU-worth of serving capacity. Your capacity planning must account for this: always run with enough headroom that losing one parallelism group does not cause overload on the remaining groups.

Throughput optimization can obscure quality problems. A system that processes 10,000 tokens per second but generates incorrect or hallucinated outputs has high throughput but low value. Always measure output quality alongside throughput in your optimization loops. Techniques like quantization and pruning that improve throughput can degrade quality in subtle ways that only surface on specific input distributions.

The most dangerous failure mode is when a quantization scheme passes offline quality evaluations but regresses on production inputs. Offline evaluations use benchmark datasets that may not cover the long tail of unusual queries that real users submit. A model quantized to INT4 might score within 1% of the full-precision model on standard benchmarks but fail noticeably on mathematical calculations, rare vocabulary, or code in unusual programming languages.

Build quality metrics into your serving pipeline so that throughput improvements do not silently ship regressions. Sampling a small fraction of production traffic (with appropriate privacy controls) for automated quality evaluation is a standard practice in mature ML serving systems. Set up alerts that trigger if quality metrics drop more than a threshold from the baseline, and be prepared to roll back throughput optimizations that turn out to cause quality regressions in production.

There is also a less-discussed cost to throughput optimization: operational complexity. A system running tensor parallelism with 4 GPUs, pipeline parallelism across 2 nodes, and a custom KV cache management layer is significantly harder to operate than a single-GPU deployment. Debugging is harder, upgrades are riskier, and on-call engineers need deeper expertise.

The operational cost is real money: not in hardware, but in engineering time. For many organizations, the right answer is to start with data parallelism on the largest GPU available, get serving working correctly, and add complexity only when simple capacity scaling becomes cost-prohibitive. The complexity tax of tensor and pipeline parallelism is worth paying when model size demands it, not before.

Throughput optimization for language models requires understanding several interlocking concepts:

• Measuring throughput correctly means distinguishing request throughput from token throughput, measuring under sustained load rather than brief bursts, reporting percentile latencies alongside throughput, and understanding the throughput-latency curve that governs all optimization decisions.
• Static batching is simple but wasteful: early-finishing requests hold batch slots idle, and variable-length inputs require padding waste. It is a poor fit for the length-variable outputs common in production, where the gap between the shortest and longest request can span two orders of magnitude.
• Continuous batching solves these problems by reassigning batch slots at every generation step. It provides 2-4x throughput improvement over static batching on mixed-length workloads and is the standard approach in modern serving frameworks. PagedAttention provides the memory management foundation that makes efficient continuous batching possible.
• Batch size tuning involves finding the point where the GPU transitions from memory-bandwidth-bound to compute-bound. The crossover batch size for a 7B model on an A100 is roughly 150. Below this threshold, larger batches dramatically improve throughput by amortizing weight-read costs. Above it, the gains diminish.
• Parallelism strategies scale beyond a single GPU. Data parallelism provides linear scaling with no communication overhead but requires the full model on each GPU. Tensor parallelism splits layers across GPUs with all-reduce overhead, allowing models that exceed single-GPU memory. Pipeline parallelism assigns layers to GPU stages with bubble overhead, scaling across nodes. Production deployments combine all three based on model size and hardware topology.
• Tradeoffs are unavoidable. Throughput optimization competes with latency (via batch queueing), output quality (via quantization), reliability (via parallelism failure domains), and operational simplicity (via system complexity). SLO constraints determine how each decision balances these tensions.

The next chapter covers monitoring in production, where you will learn how to observe these throughput metrics continuously and detect when serving performance degrades before it affects users.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about throughput optimization for language models.