Latency Optimization › Summary

LLM serving latency breaks down into a small number of measurable stages: queue time, prefill (which determines TTFT), and decode (which determines TPOT). Each stage has a different bottleneck and a different set of optimizations.

The key insights are:

• Prefill is compute-bound and scales with prompt length. Prefix caching eliminates redundant prefill for shared prompt prefixes. The KV cache must be carefully sized and managed, because at long contexts it dominates GPU memory usage.
• Decode is memory-bandwidth bound and improves with batching. Continuous batching keeps the GPU fully utilized by refilling completed sequence slots immediately. Weight quantization directly reduces the memory bandwidth required per decode step, often halving TPOT for int8 weights.
• Speculative decoding uses a fast draft model to generate candidate tokens, then verifies them in parallel with the target model, achieving exact sampling while reducing the number of full-model forward passes. It is most effective on predictable, conversational workloads where acceptance rates are high.
• Streaming decouples perceived responsiveness from total completion time by delivering tokens to the client as they are generated. It is the highest-ROI optimization for user-facing applications because it improves the subjective experience without changing any serving infrastructure.
• Profiling with per-stage breakdowns is the foundation of all latency optimization: different stages have different bottlenecks and require different solutions. Queue-dominated systems need more capacity; prefill-dominated systems need prefix caching; decode-dominated systems need batching and quantization.
• Trade-offs are unavoidable: latency optimizations often conflict with throughput, and every system must find the balance appropriate to its SLOs and workload.

Together, these techniques make it practical to deploy large language models with latencies that feel interactive rather than slow, transforming the user experience without sacrificing output quality or model capability. The next chapter explores the complementary challenge of throughput optimization, where the goal shifts from minimizing per-request latency to maximizing the number of requests served per second.