Latency Optimization › Putting It Together: A Latency Optimization Workflow
Language AI HandbookAll chapters Latency Optimization › Putting It Together: A Latency Optimization WorkflowOne section · Sections in this chapter8 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
A principled approach to latency optimization follows a consistent sequence of steps regardless of the specific system.
The first step is measurement. Before changing anything, instrument the serving system to collect TTFT, TPOT, queue time, and per-stage breakdowns. Establish baseline percentiles (P50, P95, P99) for your target workload. This baseline is what all subsequent improvements will be compared against.
The second step is identifying the dominant bottleneck. Examine the latency breakdown. If queue time dominates, the problem is capacity, not optimization. If prefill time is high relative to prompt length, investigate whether prefix caching is effective. If TPOT is high and not improving with higher batch sizes, the model may be too large for the available hardware and quantization or model selection may be the right lever.
The third step is applying targeted optimizations. Continuous batching is almost always worth enabling if not already in place. Prefix caching is effective for workloads with repetitive system prompts. Speculative decoding provides the most benefit on workloads where the draft acceptance rate is high (conversational tasks tend to have higher acceptance rates than creative writing tasks because the vocabulary and style are more predictable). Streaming should be enabled for all user-facing applications regardless of other optimizations.
The fourth step is re-measuring and verifying that the optimization improved the right metric without degrading others. Speculative decoding, for instance, can increase per-request compute and memory usage, potentially hurting throughput. The optimization is only beneficial if the latency improvement is worth the throughput cost for your specific SLOs.
Beyond these four core steps, it is worth considering the interaction between hardware and optimization strategy. A serving stack on older hardware with lower memory bandwidth may benefit more from quantization than one on the latest A100 or H100 GPUs. A system that runs on many small GPUs connected over NVLink may see different bottlenecks from one running on a single large GPU. The optimizations described in this chapter are general principles; their relative importance varies with hardware, model size, and workload characteristics.