Latency Optimization › Latency Optimization

When you type a prompt and wait for a language model to respond, time passes across a surprising number of stages. There is the moment the request travels to the server, the setup the model does before it starts generating, the time spent producing each token, and finally the trip your answer takes back across the network. Each of these stages contributes to the total wall-clock time you experience, yet they have almost nothing to do with one another. Reducing one does not automatically reduce the others. That asymmetry is the central insight of latency optimization: you must understand which stage governs the user experience before you can make meaningful improvements.

Latency matters in a way that raw throughput does not. A model that produces a thousand tokens per second across a large batch might still feel sluggish if it takes five seconds before the first word appears. Conversely, a model that starts writing in under a second feels responsive even when the full completion takes longer. This chapter dissects the latency budget of a modern LLM serving system, explains the bottlenecks that arise at each stage, and covers the practical strategies, from KV-cache management to speculative decoding to streaming, that practitioners use to make large models feel fast.

What makes LLM latency optimization unusual compared to traditional software performance work is the interplay between memory, compute, and user perception. Classical web services are often network-bound or database-bound; optimize the database query and the page loads faster. LLM serving combines memory-bandwidth bottlenecks during decode, compute bottlenecks during prefill, and psychological factors around what humans perceive as "fast." A response that streams words at a natural reading pace feels interactive even when the underlying model is not unusually fast. Engineering for perceived latency is as important as engineering for measured latency.

There is also an important distinction between latency and throughput that shapes every design decision in this chapter. Throughput measures how many requests a system can serve per unit of time across a whole workload. Latency measures how long a single user waits for their answer. Many techniques that improve throughput (larger batches, longer queues) hurt individual latency. The techniques in this chapter are specifically chosen to improve latency, and we will be explicit about the throughput trade-offs they introduce.