Throughput Optimization › Limitations and Practical Considerations

Throughput optimization is powerful, but it is not free. Understanding the real-world constraints prevents you from running into surprises in production.

The KV cache is the central resource constraint. For long-context models (128K tokens), the KV cache for a single request can exceed the size of the model weights. A 70B model in FP16 uses about 140 GB for weights, but a single 128K-token request in that model uses roughly 64 GB of KV cache. This makes large-batch serving of long-context requests nearly impossible on current hardware without aggressive KV cache compression, quantization, or eviction strategies.

Systems that look excellent in benchmarks with 256-token contexts can fail catastrophically when users submit long documents. A serving system designed for short contexts will hit out-of-memory errors the first time a user submits a 10,000-token document for summarization, even if the model technically supports that context length. You must design your memory management to handle the full range of context lengths that your users will submit, including outliers.

The practical mitigations are:

• KV cache quantization: Store KV entries in INT8 or FP8 instead of FP16, halving memory usage with typically negligible quality loss on most tasks.
• KV cache eviction: When memory is full, evict the KV cache entries for the least recently accessed token positions and recompute them on demand. This trades compute for memory and is expensive for frequently accessed positions.
• Context length limits: Cap the context length at a value your hardware can handle at your target batch size. This is the simplest solution and is appropriate when you know users will not submit very long inputs.
• Streaming prefill: Process very long prompts in chunks, holding only the most recent chunk's activations in memory at once. This reduces peak memory during prefill but does not reduce decode-time KV cache size.

Continuous batching introduces a subtler limitation: head-of-line blocking from very long requests. A request that runs for 2,000 decode steps occupies a batch slot for the entire duration. Other requests that arrive during those 2,000 steps must compete for the remaining batch slots. If several long requests arrive simultaneously and fill the batch, short requests queue behind them even though they would complete in 10 steps.

In practice, you mitigate this by setting a maximum output length policy and by using preemption mechanisms (checkpointing and restarting long requests) in extreme cases. Maximum output length is the simpler solution: truncate generation at some maximum (e.g., 2,048 tokens) and return whatever the model has generated so far. Users who need longer outputs must chunk their requests. Preemption is more complex, requiring the scheduler to checkpoint the long request's KV cache state, free its batch slot, and restore it later. Most production serving frameworks implement maximum output length; preemption is less common because of the operational complexity.

The throughput gains from parallelism strategies come with reliability implications. A tensor-parallel inference group across 8 GPUs fails completely if any one GPU goes down. Data parallelism provides natural fault tolerance (the other replicas continue serving), while tensor and pipeline parallelism create correlated failure domains. In high-availability deployments, you typically run at least two replicas of each parallelism group so that one failure does not take down serving.

The mean time between failures (MTBF) for a single A100 GPU is on the order of 1-3 years, but a cluster of 512 GPUs expects a failure roughly every day. For a deployment using 8-GPU tensor-parallel groups, a single GPU failure takes down 8 GPU-worth of serving capacity. Your capacity planning must account for this: always run with enough headroom that losing one parallelism group does not cause overload on the remaining groups.

Throughput optimization can obscure quality problems. A system that processes 10,000 tokens per second but generates incorrect or hallucinated outputs has high throughput but low value. Always measure output quality alongside throughput in your optimization loops. Techniques like quantization and pruning that improve throughput can degrade quality in subtle ways that only surface on specific input distributions.

The most dangerous failure mode is when a quantization scheme passes offline quality evaluations but regresses on production inputs. Offline evaluations use benchmark datasets that may not cover the long tail of unusual queries that real users submit. A model quantized to INT4 might score within 1% of the full-precision model on standard benchmarks but fail noticeably on mathematical calculations, rare vocabulary, or code in unusual programming languages.

Build quality metrics into your serving pipeline so that throughput improvements do not silently ship regressions. Sampling a small fraction of production traffic (with appropriate privacy controls) for automated quality evaluation is a standard practice in mature ML serving systems. Set up alerts that trigger if quality metrics drop more than a threshold from the baseline, and be prepared to roll back throughput optimizations that turn out to cause quality regressions in production.

There is also a less-discussed cost to throughput optimization: operational complexity. A system running tensor parallelism with 4 GPUs, pipeline parallelism across 2 nodes, and a custom KV cache management layer is significantly harder to operate than a single-GPU deployment. Debugging is harder, upgrades are riskier, and on-call engineers need deeper expertise.

The operational cost is real money: not in hardware, but in engineering time. For many organizations, the right answer is to start with data parallelism on the largest GPU available, get serving working correctly, and add complexity only when simple capacity scaling becomes cost-prohibitive. The complexity tax of tensor and pipeline parallelism is worth paying when model size demands it, not before.