LLM Inference Serving: Architecture, Routing & Auto-Scaling › Latency Optimization
Language AI HandbookAll chapters LLM Inference Serving: Architecture, Routing & Auto-Scaling › Latency OptimizationOne section · Sections in this chapter6 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Meeting latency targets requires understanding the distinct phases of LLM inference and optimizing each appropriately. Because the prefill and decode phases have fundamentally different computational characteristics, they require different optimization strategies. A one-size-fits-all approach will inevitably leave performance on the table.
Think of optimizing LLM latency as optimizing a two-stage pipeline where each stage has a completely different bottleneck. The prefill stage is like a high-throughput assembly line that can process many units in parallel: the more parallelism you add (more GPUs, larger matrix operations), the faster it goes, and the bottleneck is arithmetic throughput. The decode stage is like a process that must read from a very slow shared library for every single item: the bottleneck is how quickly you can access memory, and adding more compute units helps very little because they all spend most of their time waiting for memory reads to complete. Recognizing these distinct bottlenecks is the first step toward effective optimization.
The practical implication is that profiling your specific workload matters enormously. A chatbot with short system prompts and short responses will spend most of its time in decode, so KV cache compression and speculative decoding are high-value investments. A document summarization service with 3000-token prompts and 200-token summaries will spend a large fraction of its time in prefill, so chunked prefill and tensor parallelism are more valuable. No single optimization profile fits all workloads, and measuring where your time goes prevents you from over-investing in optimizations that target the wrong bottleneck.
Prefill vs Decode Optimization
As we have seen throughout this part, LLM inference has two distinct phases with different performance characteristics that demand different optimization approaches.
The prefill phase involves processing the input prompt by computing attention across all prompt tokens simultaneously. This phase is compute-bound because it performs dense matrix multiplications with operations proportional to the square of the prompt length (due to attention) times the model dimension. The GPU's compute units are the bottleneck, and memory bandwidth is typically not limiting. Optimization strategies for the prefill phase include:
- Chunked prefill: Split the prefill phase into chunks to avoid blocking the decode phase of other requests, letting interleaved progress on multiple requests
- Tensor parallelism: Distribute compute across GPUs, reducing wall-clock time for long prompts
- Quantized attention: Faster matrix operations that trade some precision for speed
The decode phase generates output tokens one at a time, meaning each forward pass produces only a single token while reading the entire KV cache from memory. This phase is memory-bandwidth-bound because the ratio of computation to memory access is low. Each token requires reading billions of bytes of model weights and cached attention states while performing relatively few arithmetic operations. Optimization strategies for the decode phase include:
- KV cache compression: Reduce memory reads by shrinking the data that must be transferred each step
- Speculative decoding: Generate multiple tokens per forward pass, amortizing memory access cost
- Continuous batching: Share the memory bandwidth cost across multiple requests, improving effective utilization
# Simulate timing for different request profiles
# Assumptions: 1000-token prompt, varying output lengths
# Prefill: ~0.1ms per token (compute-bound)
# Decode: ~20ms per token (memory-bound)
prompt_tokens = 1000
prefill_time_per_token = 0.1 # ms
decode_time_per_token = 20 # ms
output_lengths = [10, 50, 100, 200, 500]
prefill_times = [
prompt_tokens * prefill_time_per_token / 1000 for _ in output_lengths
] # seconds
decode_times = [
out * decode_time_per_token / 1000 for out in output_lengths
] # seconds
For a typical prompt of 1000 tokens, prefill completes in roughly 100ms. A 10-token response adds 200ms of decode time, making prefill a significant fraction (33%) of total latency. A 500-token response adds 10 seconds of decode time, making prefill negligible (less than 1%). This dramatic shift means optimization priorities should align with your actual request distribution. If most requests generate short outputs, optimizing prefill matters more. If most requests generate long outputs, decode optimization dominates.
Time-to-First-Token Optimization
Many applications stream responses to users, making time-to-first-token (TTFT) the primary latency metric that determines perceived responsiveness. When a user sends a message in a chat interface, TTFT measures the time from submission until the first word of the response appears. Users perceive systems with low TTFT as fast and responsive, even if total generation time is similar.
TTFT matters psychologically as much as technically. Research on user experience shows that users form lasting impressions of a system's speed based heavily on how quickly it begins responding, not just how long it takes to complete. A system that starts generating immediately and takes 5 seconds total feels faster than one that waits 4 seconds before generating anything and completes in 3 seconds total, even though the second system is technically faster. This means that even if total token generation speed cannot be improved, reducing TTFT can improve users' perception of system quality.
TTFT measures the time from request arrival until the first generated token is available to the client. This encompasses several distinct stages, each giving optimization opportunities:
- Queue Time: How long the request waits before processing begins. Reducing queue time requires either more capacity (scaling) or smarter scheduling (prioritization).
- Tokenization: Converting the prompt text to token IDs. While typically fast, this step runs on CPU and can bottleneck under high load.
- Prefill Duration: Processing all prompt tokens to populate the KV cache. For long prompts, this dominates TTFT.
- First Decode Step: Generating the first output token. This is typically fast since it's a single forward pass.
from dataclasses import dataclass
from typing import List
import numpy as np
@dataclass
class TTFTBreakdown:
"""Breakdown of time-to-first-token components."""
queue_time_ms: float
tokenization_ms: float
prefill_ms: float
first_decode_ms: float
@property
def total_ms(self) -> float:
return (
self.queue_time_ms
+ self.tokenization_ms
+ self.prefill_ms
+ self.first_decode_ms
)
def bottleneck(self) -> str:
components = {
"queue": self.queue_time_ms,
"tokenization": self.tokenization_ms,
"prefill": self.prefill_ms,
"decode": self.first_decode_ms,
}
return max(components, key=components.get)
def analyze_ttft_samples(samples: List[TTFTBreakdown]) -> dict:
"""Compute TTFT statistics across samples."""
totals = [s.total_ms for s in samples]
bottlenecks = [s.bottleneck() for s in samples]
return {
"p50_ms": np.percentile(totals, 50),
"p95_ms": np.percentile(totals, 95),
"p99_ms": np.percentile(totals, 99),
"mean_ms": np.mean(totals),
"bottleneck_distribution": {
b: bottlenecks.count(b) / len(bottlenecks) for b in set(bottlenecks)
},
}# Simulate TTFT measurements under different load conditions
# Low load scenario
low_load_samples = [
TTFTBreakdown(
queue_time_ms=np.random.exponential(5),
tokenization_ms=np.random.normal(10, 2),
prefill_ms=np.random.normal(100, 20),
first_decode_ms=np.random.normal(20, 5),
)
for _ in range(100)
]
# High load scenario
high_load_samples = [
TTFTBreakdown(
queue_time_ms=np.random.exponential(200), # Much higher queue times
tokenization_ms=np.random.normal(10, 2),
prefill_ms=np.random.normal(100, 20),
first_decode_ms=np.random.normal(20, 5),
)
for _ in range(100)
]
low_load_stats = analyze_ttft_samples(low_load_samples)
high_load_stats = analyze_ttft_samples(high_load_samples)TTFT Analysis:
--------------------------------------------------
Low Load:
P50: 138ms
P95: 165ms
P99: 183ms
Bottleneck distribution:
prefill: 100%
High Load:
P50: 248ms
P95: 719ms
P99: 1339ms
Bottleneck distribution:
prefill: 42%
queue: 58%
The analysis reveals how bottlenecks shift with load. Under low load, prefill dominates TTFT because queue times are minimal and requests begin processing immediately upon arrival. Optimizing prefill speed through better hardware or chunking would improve TTFT in this regime. Under high load, queue time becomes the dominant bottleneck: requests spend most of their time waiting, not processing. In this regime, improving prefill speed won't help much because requests are already waiting; adding capacity or improving scheduling would be more effective. This analysis guides optimization investments toward the actual bottleneck rather than optimizing components that aren't limiting.
Latency-Throughput Trade-offs
Optimizing for latency often conflicts with optimizing for throughput because the techniques that improve one typically harm the other. Larger batches improve GPU utilization and throughput: processing 32 requests in one batch uses GPU compute more efficiently than processing them one at a time. However, larger batches increase latency for individual requests because each request waits for the batch to form and then shares compute time with other requests in the batch.
The key insight is that latency and throughput are not independent dimensions that can be independently optimized; they are coupled through the shared resource of GPU compute. The basic relationship can be stated simply: throughput is rate (requests per second) and latency is time per request. When requests share compute in a batch, each request takes longer (higher latency) but the aggregate rate increases (higher throughput). There is no way to have both maximum throughput and minimum latency simultaneously at high concurrency.
The optimal operating point depends on your SLA requirements and cost constraints:
import numpy as np
# Simulated relationship between batch size and performance
batch_sizes = np.array([1, 2, 4, 8, 16, 32, 64])
# Throughput increases with batch size but diminishes
# GPU utilization improves until we hit memory/compute limits
throughput = 50 * np.log2(batch_sizes + 1) # tokens/sec
throughput = np.minimum(throughput, 300) # Cap at 300 tok/s
# Latency increases linearly with batch size (simplified model)
# Real relationship is more complex due to batching and scheduling
base_latency = 100 # ms for batch_size=1
latency = base_latency + batch_sizes * 15 # ms
If your SLA requires P95 latency under 500ms, you're constrained to batch sizes below approximately 32 in this scenario. Larger batches would improve throughput (and thus cost efficiency) but violate the latency requirement. The continuous batching techniques from the previous chapter ease this trade-off by letting different requests enter and exit batches dynamically, improving throughput while limiting how long any individual request must wait.