Latency Optimization › Understanding the Latency Budget

A request to an LLM serving system travels through a pipeline of stages, and the total latency is the sum of all of them. Understanding this pipeline precisely is the first step toward optimizing it, because different optimizations target different stages.

The stages and the metrics that matter most for each one are worth enumerating clearly.

Time to first token (TTFT) is the elapsed time from when a client sends a request to when the first token of the response arrives. This metric governs perceived responsiveness. If you are building a chat interface, TTFT is the most user-visible latency metric, because it determines how long the screen remains blank after the user submits a message.

TTFT is dominated by the prefill phase: the forward pass over the entire input prompt. During prefill, the model processes all $N$ input tokens simultaneously, computing keys and values for every layer and populating the KV cache. This work scales with the prompt length. A 4,000-token system prompt is dramatically more expensive to prefill than a 50-token conversational message. Improving TTFT typically means either reducing prompt length, speeding up the prefill computation (for example with tensor parallelism), or caching prefill results so they do not have to be recomputed.

One subtlety that often surprises practitioners is that TTFT also includes queue time: the interval between when a request arrives and when the model begins processing it. On a lightly loaded system, queue time is negligible. On a saturated system, a request might wait hundreds of milliseconds before prefill even begins. This means that TTFT SLOs can be violated not because prefill is slow, but because the serving system is overloaded. Optimizing prefill time on an overloaded system does nothing for user experience; adding capacity or better request scheduling is the correct lever.

Serving systems often distinguish between cold TTFT (the first request after a cold start, when nothing is cached) and warm TTFT (subsequent requests that benefit from prefix caching and warm GPU caches). In production, you care primarily about warm TTFT under steady load, but cold TTFT matters for auto-scaling scenarios where new instances are spun up to handle load spikes.

Time per output token (TPOT), also written as inter-token latency, is the time between consecutive generated tokens. It determines how fast the streamed response visually updates. If TPOT exceeds roughly 60 to 80 milliseconds per token, a human reader will notice the model "typing" one word at a time at a pace that feels slow. At 50 ms per token, the experience resembles a fast typist. At 30 ms or below, the text appears nearly instantaneous.

The decode phase that governs TPOT is fundamentally different from prefill. During decoding, the model generates tokens one at a time, each requiring a full forward pass over the model, but reusing the stored KV cache from previous steps. Each decode step processes only a single token (or a small batch), so the computation is relatively light, but it still reads the entire set of model weights from memory. On most hardware, decode is memory-bandwidth bound, not compute bound. The limiting factor is how fast the GPU can stream weights from HBM (high-bandwidth memory) into the compute cores, not how fast it can multiply matrices.

The implication is that arithmetic intensity (the ratio of floating-point operations to memory bytes transferred) during decode is very low. A transformer with 70 billion parameters at 16-bit precision requires moving roughly 140 GB of weights per decode step. If the GPU provides 2 TB/s of memory bandwidth, the raw minimum decode time is 70 ms per step per request. Batching multiple requests together increases arithmetic intensity and amortizes that memory read across many outputs simultaneously. This single observation explains most of the architecture of modern LLM serving systems: the entire engineering effort around batching, continuous scheduling, and speculative decoding is, at root, an effort to overcome the memory-bandwidth bound on decode.

TPOT is not perfectly constant across a generation. The first few decode steps after prefill may be slower because the GPU is transitioning from the compute-intensive prefill kernel to the memory-bandwidth-intensive decode kernel. Steps that trigger KV cache management operations (reordering pages, evicting less-used entries) are also slower. Production profiling typically shows a distribution of per-step TPOT values rather than a single constant, and the P99 of that distribution can be significantly higher than the median.

End-to-end latency is the time between sending a request and receiving the complete response. It is approximately:

$$\text{latency}_{\text{e2e}} \approx \text{TTFT} + (\text{TPOT} \times N_{\text{out}})$$

where $N_{\text{out}}$ is the number of output tokens generated.

When streaming is disabled (the client waits for the full completion), end-to-end latency is the only metric that matters. When streaming is enabled, TTFT and TPOT are both important, since users see each token arrive incrementally.

A useful reframing of this equation is to ask what drives end-to-end latency across different workload types. For a short conversational response of 50 tokens, TTFT may account for the majority of latency. For a long document generation of 2,000 tokens, the decode term dominates entirely. Optimizing TTFT for a long-generation workload has diminishing returns because decode time is so much larger; the reverse is true for short-answer tasks.

To see why decode is memory-bandwidth bound, consider the arithmetic intensity of a single matrix-vector product, which is the dominant operation during a single-token decode step. For a weight matrix $W \in \mathbb{R}^{d \times d}$ and input vector $x \in \mathbb{R}^d$:

• FLOPs: $2d^2$ multiply-add operations
• Bytes read: $2d^2$ (for the weight matrix, at 2 bytes per float16 element) plus $2d$ (for the input vector)

The arithmetic intensity is approximately:

$$\text{AI} = \frac{2d^2}{2d^2 + 2d} \approx 1 \text{ FLOP/byte}$$

Modern GPUs can perform roughly 100-300 FLOPs per byte of memory bandwidth (their compute-to-bandwidth ratio). When arithmetic intensity is only 1 FLOP/byte, the GPU's compute units sit idle while waiting for memory. Increasing batch size during decode raises arithmetic intensity because the same weight matrix serves multiple input vectors simultaneously, moving the operation closer to the hardware's compute-bound regime.

This analysis also explains why quantization has such a significant impact on decode speed. If you represent weights at int8 instead of float16, the same weight data occupies half the memory. A matrix that previously required reading 140 GB of bytes from HBM now requires only 70 GB, and the minimum decode latency halves accordingly. The arithmetic operations are still performed at higher precision after dequantization, so quality is largely preserved. This is the core reason weight-only quantization (quantizing weights but not activations) is so popular for serving: it directly attacks the memory-bandwidth bottleneck without the accuracy risks of full quantization.

The chart makes the decode bottleneck concrete. A single-request decode step achieves barely 1 FLOP per byte of memory bandwidth transferred, while the hardware can sustain 100 or more FLOPs per byte when kept busy. Prefill over a 512-token prompt sits comfortably in the compute-bound regime because all those tokens are processed in a single large matrix multiplication. The key takeaway is that even a batch of 64 concurrent decode requests barely scratches the roofline, which is why practitioners focus so heavily on maximizing batch sizes during decode.

Beyond GPU computation, requests spend time in queuing (waiting for the model to become available), serialization (converting tensors to bytes for transmission), and network transit. In collocated deployments these overheads are typically under 10 milliseconds, but in multi-region or edge deployments network round-trip time can dominate. A round trip from Europe to a US-East data center adds roughly 100 ms of irreducible latency that no amount of GPU optimization can remove. Latency profiling must instrument all stages to avoid optimizing the wrong part of the pipeline.

Queuing deserves special attention because it is often the dominant component in production systems under load. When request arrival rate approaches service rate, queuing theory predicts that average queue time grows without bound. Even at 80% utilization, a system with Poisson arrivals experiences average queue times roughly four times the average service time. This means that a system with 500 ms average service time at 80% utilization will have average queue times of 2,000 ms, making TTFT nearly 2.5 seconds regardless of how fast the GPU is. Capacity planning and admission control are therefore first-order latency concerns, not afterthoughts.