Latency Optimization › Latency Profiling
Language AI HandbookAll chapters Latency Optimization › Latency ProfilingOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Optimizing latency without measuring it is guesswork. A profiling system that instruments every stage of the request pipeline turns optimization from art into engineering.
Profiling an LLM serving system is more challenging than profiling a typical web service because the stages of interest span multiple levels of abstraction: Python application code, CUDA kernels on the GPU, and hardware-level memory transactions. A request that appears slow might be waiting in a Python-level queue, waiting for a CUDA kernel to launch, waiting for GPU memory bandwidth, or all three simultaneously. Effective profiling instruments each level and correlates their observations into a single per-request timeline.
The most actionable profiling data combines high-level per-stage timing (easy to collect in Python with time.monotonic()) with periodic GPU utilization and memory bandwidth metrics (available through NVIDIA's DCGM telemetry and Nsight Systems). Correlating these two streams lets you distinguish between "request is slow because it is waiting to be scheduled" and "request is slow because its GPU kernels are bottlenecked on memory bandwidth."
Key Metrics to Instrument
A well-instrumented LLM serving system tracks the following for every request:
- Queue time: time spent waiting for a free GPU slot or batch slot
- Prefill time: time to compute attention and fill the KV cache for the input prompt
- Decode time: total time across all autoregressive steps
- Per-step decode time: time for each individual decode iteration (useful for detecting slowdowns from KV cache eviction or memory pressure)
- Network time: time for the response to travel from server to client
import time
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class RequestTrace:
request_id: str
prompt_tokens: int
output_tokens: int = 0
queue_start: float = field(default_factory=time.monotonic)
queue_end: Optional[float] = None
prefill_start: Optional[float] = None
prefill_end: Optional[float] = None
decode_start: Optional[float] = None
decode_end: Optional[float] = None
@property
def queue_time_ms(self) -> float:
if self.queue_end is None:
return 0.0
return (self.queue_end - self.queue_start) * 1000
@property
def prefill_time_ms(self) -> float:
if self.prefill_start is None or self.prefill_end is None:
return 0.0
return (self.prefill_end - self.prefill_start) * 1000
@property
def decode_time_ms(self) -> float:
if self.decode_start is None or self.decode_end is None:
return 0.0
return (self.decode_end - self.decode_start) * 1000
@property
def ttft_ms(self) -> float:
"""Time from queue start to first token (end of prefill)."""
if self.prefill_end is None:
return 0.0
return (self.prefill_end - self.queue_start) * 1000
@property
def tpot_ms(self) -> float:
"""Average time per output token."""
if self.output_tokens <= 1 or self.decode_time_ms == 0:
return 0.0
return self.decode_time_ms / self.output_tokens=== Latency Profile Summary (N=200 simulated requests) === Time to First Token (TTFT): P50: 269.2 ms P95: 786.1 ms P99: 1012.6 ms Time per Output Token (TPOT): P50: 51.5 ms P95: 64.4 ms P99: 68.4 ms End-to-End Latency: P50: 14.70 s P95: 25.48 s P99: 29.95 s Latency Breakdown (median request): Queue: 21.8 ms (0.1%) Prefill: 235.4 ms (1.6%) Decode: 14419.8 ms (98.2%)
The profile shows how latency is distributed across stages and that different stages dominate for different workloads. Short requests with long system prompts are prefill-dominated; long generation tasks are decode-dominated; overloaded systems are queue-dominated. This three-way breakdown is the starting point for any optimization effort: there is no universal best strategy, only strategies that are best for the specific breakdown you observe.
The contrasting shapes of these two distributions capture the essential asymmetry of LLM latency. TTFT has a heavy right tail driven by long prompts, while TPOT is tightly clustered because it is hardware-bound rather than workload-bound. This asymmetry matters for SLO design: TTFT P99 can be much larger than P50, requiring the tail to be specifically optimized (for example by prioritizing short-prompt requests), while TPOT can be reliably predicted from hardware specifications.
Identifying Bottlenecks
With per-stage latency data, bottleneck identification becomes systematic. A few patterns are particularly actionable:
- High queue time relative to total latency indicates capacity shortage. The system is overloaded and either needs more capacity or better request admission control.
- Prefill time growing faster than prompt length suggests memory fragmentation or lack of prefix caching. A 2x increase in prompt tokens should produce at most a 2x increase in prefill time.
- TPOT varying with batch size confirms memory-bandwidth-bound behavior. If TPOT increases as batch size grows beyond a threshold, the GPU memory bandwidth is saturated and the batch is too large.
- Outlier TPOT values (P99 >> P50) often indicate KV cache eviction events where the system is swapping cached states to CPU memory to handle new requests.
The scatter plot confirms the linear relationship between prompt length and prefill time. The slope, roughly 0.5 ms per token, is the simulation's effective per-token prefill cost. Requests that fall well above the trend line are candidates for optimization via prefix caching or prompt compression.
Production Monitoring Infrastructure
Beyond per-request profiling, latency optimization requires continuous production monitoring. Latency regressions can be introduced silently by changes to model versions, serving configuration, hardware utilization trends, or traffic pattern shifts. A monitoring system that tracks latency percentiles over time, alerts on P95 threshold violations, and provides dashboards for capacity planning is essential infrastructure for any serious LLM deployment.
The metrics that matter most for production monitoring are organized around the percentiles that correspond to your SLOs. If your TTFT SLO is P95 under 1 second, you need a timeseries of P95 TTFT measured every minute, with alerts when it exceeds threshold. Tracking only averages is insufficient: the average TTFT can look healthy while P99 is already degrading, which is the situation that causes user complaints.
Distributed tracing (using systems like OpenTelemetry, Jaeger, or cloud-native equivalents) provides the full picture by correlating traces across all layers of the serving stack. Each request gets a unique trace identifier that propagates through all components, allowing you to reconstruct the complete timeline of a slow request including queue time, model initialization, GPU kernel launches, and network transmission. This is invaluable for diagnosing tail-latency issues that only appear on specific request types or under specific load conditions.