Throughput Optimization › Throughput Tradeoffs

Every throughput optimization involves tradeoffs. Understanding them prevents optimizing one dimension at the cost of another. In production systems, you will regularly face decisions where the highest-throughput configuration is not the right configuration, because it sacrifices latency, quality, or reliability in ways your product cannot accept.

The most fundamental tradeoff is between throughput and latency. To maximize throughput, you want large batches: the GPU is always busy, and each weight-read is amortized over many requests. But large batches mean requests wait in queue for a batch slot, increasing latency.

Several levers control where you sit on this tradeoff:

• Maximum batch size: The hard ceiling on how many requests can be in flight simultaneously. Higher values increase throughput but can spike latency under variable load.
• Maximum queue time: How long a request waits before being processed even with an incomplete batch. Lower values reduce latency but allow smaller, less efficient batches.
• Priority tiers: Some systems maintain separate queues for interactive and batch workloads, allowing different latency targets.

The practical implication is that your SLO sets a ceiling on batch size. If your P99 latency SLO is 500 ms and your model takes 5 ms per decode step, you can afford at most 100 decode steps worth of queueing. At 5 ms per step per batch of 32 requests, this constrains how long requests wait. You then select the largest batch size that keeps you within this constraint.

Different product types have radically different SLO profiles. An interactive chat interface requires P99 latency under 2 seconds to feel responsive. A code completion system in an IDE requires P99 latency under 200 ms to not interrupt the developer's flow. A document summarization pipeline running overnight can tolerate latency of minutes, making throughput optimization essentially unconstrained by latency concerns. Always clarify the SLO before optimizing.

Quantization reduces memory footprint and increases arithmetic intensity, enabling larger batches and higher throughput. But lower-precision arithmetic (INT8, INT4) can degrade output quality, particularly on tasks requiring precise reasoning or mathematical computation.

This tradeoff is workload-dependent. Customer service chatbots may tolerate INT4 quantization with negligible quality loss, while scientific reasoning tasks may show measurable degradation. Always evaluate quality alongside throughput when choosing a quantization scheme. A common framework is to measure quality on a representative held-out set before and after quantization, computing the quality delta relative to the throughput improvement. If INT8 quantization improves throughput by 40% at a quality delta of 0.5%, that is usually an excellent trade. If it improves throughput by 20% at a quality delta of 5%, you need to evaluate whether your use case can absorb that quality loss.

Speculative decoding (covered in the Latency Optimization chapter) provides a different tradeoff: it typically maintains full output quality (because the large model verifies every token) but uses extra compute on speculative drafts. For throughput-focused serving, speculative decoding is usually less beneficial than batching improvements, because it adds overhead per request rather than amortizing costs across requests.

A recent architectural insight is that prefill and decode have fundamentally different compute characteristics and should be handled separately.

Prefill (processing the input prompt) is compute-intensive: you process all input tokens in parallel, which maps well to GPU matrix operations. A long prompt (1,000 tokens) benefits from the full parallelism of the transformer. Arithmetic intensity during prefill is proportional to prompt length, so even single-request prefill saturates the GPU's compute units on long prompts.

Decode (generating output tokens one at a time) is memory-bandwidth-bound: you generate a single token per step, so GPU utilization is low unless you batch many requests together. As we analyzed earlier, you need batch sizes in the hundreds to approach the A100 roofline during decode.

Running both on the same GPU creates contention. Large batches optimize decode (high GPU utilization from many concurrent requests) but create unpredictable latency spikes when a long prefill interrupts decode. When a long prefill arrives on a GPU running a large decode batch, the entire batch must pause or the prefill must wait. Either outcome degrades latency: either the existing decode users experience a pause, or the new user waits unusually long to see their first token.

Disaggregation separates these concerns:

• Prefill servers: Highly capable GPUs (or multi-GPU nodes) that handle prompt processing. These see bursty, compute-heavy workloads and benefit from fast NVLink interconnects that accelerate tensor parallelism.
• Decode servers: Memory-bandwidth-optimized hardware (or commodity GPUs) that handle generation. These benefit from stable, high-concurrency batching and can use more memory-efficient hardware (like older A100s with less compute but adequate bandwidth).

Disaggregation adds operational complexity (you need two fleets with a routing layer between them) but enables better resource matching and eliminates the prefill-decode contention that degrades throughput in mixed workloads. Research systems like DistServe and Splitwise have demonstrated that disaggregation can improve goodput (the fraction of GPU time spent on useful work) by 30-50% compared to co-located serving on the same hardware.

The transfer of KV cache between prefill and decode servers is the main technical challenge. After prefill completes on the prefill server, the KV cache must be transferred to the decode server that will handle generation. This transfer uses the same InfiniBand interconnect as pipeline parallelism communication. For long prompts (thousands of tokens), the KV cache transfer can be substantial, and the routing layer must account for transfer time when making scheduling decisions.