Latency Optimization › Limitations and Trade-offs

Latency optimization in LLM serving requires trade-offs between competing objectives, and practitioners must account for what they give up when optimizing for speed.

The most persistent tension is between latency and throughput. Batching improves throughput by amortizing the memory bandwidth cost of weight loading across many requests, but larger batches mean each individual request waits longer for its slot in the batch. Continuous batching mitigates this by filling slots immediately, but a system running at high throughput will still queue requests during load spikes. Setting the right batch size limit requires knowing your latency SLOs and your expected request arrival rate simultaneously. There is no configuration that simultaneously minimizes per-request latency and maximizes system throughput; the right balance depends on whether you are building an interactive product (where latency matters most) or a batch processing pipeline (where throughput matters most).

Speculative decoding introduces a different trade-off. It reduces latency per token but increases total compute because the draft model runs on tokens that may be rejected. Under heavy load, this extra compute can reduce throughput. Speculative decoding is most beneficial when the system is compute-light (a single user chatting with a deployed model) and least beneficial when the system is fully saturated (a batch processing job at maximum throughput). Measuring both metrics before and after enabling speculative decoding is essential because the benefit varies substantially by workload.

KV cache management presents its own constraints. Larger KV caches support longer contexts but consume memory that could otherwise support larger batch sizes. Systems with tight memory budgets must choose between supporting long contexts for fewer requests or short contexts for more requests. Paged attention and quantizing the KV cache to lower precision (int8 or even int4) extend the effective memory budget, but KV quantization introduces approximation error that can affect quality at long contexts. The error is generally larger for very long contexts where the model must aggregate information from distant positions, precisely the use case that requires the longest context windows.

Streaming, while almost universally positive for perceived responsiveness, requires the client infrastructure to support incremental delivery. HTTP/1.1 connections with slow clients can create head-of-line blocking where a slow consumer holds a server-side buffer open, consuming memory and potentially blocking other requests. Production serving systems must implement client timeouts and back-pressure mechanisms to prevent this. HTTP/2 and HTTP/3 handle streaming more gracefully through stream multiplexing, which is one reason they are preferred for high-scale LLM serving deployments.

Finally, prefix caching only helps when prompts share prefixes. A system serving highly diverse, user-specific prompts may see near-zero cache hit rates, making prefix caching a source of memory overhead with no corresponding benefit. Measuring cache hit rate is essential before committing resources to a large prefix cache. In deployments where cache hit rate is low, the memory would be better used to support larger batches during decode.

There is also an infrastructure-level trade-off between operational simplicity and latency optimality. A system with continuous batching, paged attention, prefix caching, speculative decoding, and chunked prefill is a highly capable serving stack, but it is also a complex one with many interacting parameters. Each optimization introduces potential failure modes: a prefix cache eviction bug can silently serve incorrect outputs, a speculative decoding implementation bug can subtly alter output distributions, and KV quantization errors can be context-length-dependent and hard to detect in standard benchmarks. Adopting mature, well-tested serving frameworks (vLLM, TGI, TensorRT-LLM) rather than building from scratch is usually the right call for most teams, both for reliability and because these frameworks incorporate years of hard-won optimizations.