LLM Inference Serving: Architecture, Routing & Auto-Scaling › Monitoring and Observability
Language AI HandbookAll chapters LLM Inference Serving: Architecture, Routing & Auto-Scaling › Monitoring and ObservabilityOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Production inference systems require complete monitoring to maintain service quality and debug issues. Without visibility into system behavior, operators cannot distinguish between capacity problems, model issues, and infrastructure failures. Good observability enables proactive intervention before users experience degraded service.
Think of monitoring as the nervous system of your serving infrastructure. Without it, the infrastructure is operationally blind, it cannot feel pain when things go wrong, cannot detect when it is being stressed beyond its limits, and cannot signal for help before failures cascade. With good monitoring, operators get early warning of developing problems, data for capacity planning decisions, and the detailed traces needed to diagnose the root cause of any issue that does arise.
Effective monitoring operates at multiple time scales. Real-time dashboards showing current queue depth and latency percentiles let operators detect immediate problems. Hourly trend charts reveal developing capacity issues before they become necessary. Daily and weekly aggregates inform capacity planning and SLA compliance reporting. Each time scale requires different data retention and aggregation strategies, and building all of them into the monitoring infrastructure from the start is much easier than adding them after a production incident reveals their absence.
A common failure mode in monitoring infrastructure is collecting the wrong metrics rather than no metrics at all. GPU utilization sounds informative but can be misleading for LLM serving, as discussed earlier. Request count sounds basic but misses the wide variance in request cost. Building metrics around the actual resource constraints (memory bandwidth, KV cache occupancy, queue depth) rather than generic infrastructure metrics gives operators a much more accurate picture of system health.
Key Metrics
Effective monitoring tracks metrics at multiple levels, from individual request performance to aggregate system health to business outcomes.
Request-level metrics capture the experience of individual requests:
- Time to first token (TTFT): Measures perceived responsiveness for streaming applications
- Total latency: Measures end-to-end completion time for batch processing use cases
- Tokens per request (prompt and generated): Indicates request complexity and resource consumption
- Error rates by error type: Distinguishes between timeouts, out-of-memory, and other failure modes
System-level metrics reflect the health of the serving infrastructure:
- GPU utilization and memory usage: Indicates whether hardware is being used efficiently
- Queue depth over time: Reveals demand patterns and capacity mismatches
- Active batch size: Shows how effectively batching is working
- KV cache utilization: Indicates memory pressure and potential for cache eviction
Business-level metrics connect technical performance to business outcomes:
- Requests per second: Measures overall demand and system scale
- Tokens per second: Measures actual work being done, accounting for request size variation
- Cost per 1000 tokens: Combines infrastructure cost with throughput for economic analysis
- SLA compliance rate: The ultimate measure of whether the system meets its commitments
import time
from collections import deque
from dataclasses import dataclass, field
from typing import Dict
import numpy as np
@dataclass
class MetricsCollector:
"""Collects and aggregates inference metrics."""
window_seconds: float = 60.0
latency_samples: deque = field(default_factory=lambda: deque(maxlen=10000))
ttft_samples: deque = field(default_factory=lambda: deque(maxlen=10000))
token_counts: deque = field(default_factory=lambda: deque(maxlen=10000))
error_counts: Dict[str, int] = field(default_factory=dict)
def record_request(
self,
latency_ms: float,
ttft_ms: float,
prompt_tokens: int,
generated_tokens: int,
error: str = None,
):
timestamp = time.time()
self.latency_samples.append((timestamp, latency_ms))
self.ttft_samples.append((timestamp, ttft_ms))
self.token_counts.append((timestamp, prompt_tokens + generated_tokens))
if error:
self.error_counts[error] = self.error_counts.get(error, 0) + 1
def get_current_metrics(self) -> dict:
current_time = time.time()
cutoff = current_time - self.window_seconds
# Filter to recent samples
recent_latencies = [v for t, v in self.latency_samples if t > cutoff]
recent_ttft = [v for t, v in self.ttft_samples if t > cutoff]
recent_tokens = [v for t, v in self.token_counts if t > cutoff]
if not recent_latencies:
return {"status": "no_data"}
return {
"latency_p50_ms": np.percentile(recent_latencies, 50),
"latency_p95_ms": np.percentile(recent_latencies, 95),
"latency_p99_ms": np.percentile(recent_latencies, 99),
"ttft_p50_ms": np.percentile(recent_ttft, 50),
"ttft_p95_ms": np.percentile(recent_ttft, 95),
"requests_per_minute": len(recent_latencies),
"tokens_per_minute": sum(recent_tokens),
"error_counts": dict(self.error_counts),
}# Simulate metrics collection
collector = MetricsCollector(window_seconds=60)
# Simulate 200 requests over the last minute
for _ in range(200):
latency = np.random.lognormal(6, 0.5) # Log-normal distribution
ttft = np.random.lognormal(5, 0.3)
prompt_tokens = np.random.randint(100, 2000)
generated_tokens = np.random.randint(50, 500)
# 2% error rate
error = "timeout" if np.random.random() < 0.02 else None
collector.record_request(
latency, ttft, prompt_tokens, generated_tokens, error
)
metrics = collector.get_current_metrics()Current Metrics (last 60 seconds):
----------------------------------------
Requests: 200
Tokens processed: 257,565
Latency P50: 412ms
Latency P95: 940ms
Latency P99: 1320ms
TTFT P50: 150ms
TTFT P95: 259ms
Errors: {'timeout': 4}SLOs and Alerting
Service Level Objectives (SLOs) define target performance thresholds that the system should maintain. SLOs translate business requirements (users expect responsive service) into measurable technical targets (P95 latency under 5 seconds). Well-defined SLOs serve multiple purposes: they guide engineering priorities during design, provide objective criteria for evaluating incidents, and enable honest communication with customers about service commitments.
The choice of SLO thresholds reflects both technical reality and business priorities. A threshold set too tight creates constant alerts for acceptable behavior and breeds alert fatigue. A threshold set too loose provides false confidence and misses real degradation. Most teams calibrate initial thresholds based on observed performance under good conditions, then tighten them as the system matures and optimization efforts push achievable performance lower.
Common SLOs for inference services include:
- Availability: 99.9% of requests succeed (allows approximately 8.7 hours of downtime per year). This SLO accounts for both planned maintenance and unexpected failures.
- Latency: P95 latency below 5 seconds ensures that even unlucky requests complete in reasonable time.
- TTFT: P95 time-to-first-token below 500ms ensures users perceive the system as responsive.
- Error rate: Less than 0.1% of requests return errors. This keeps high reliability for applications built on the inference service.
Alerting should trigger when metrics approach SLO thresholds, not just when they're breached. Alerting only on violations means users have already experienced degraded service by the time operators learn of the problem:
@dataclass
class SLOConfig:
"""Service Level Objective configuration."""
name: str
target_percentile: float # e.g., 0.95 for P95
threshold_ms: float # SLO threshold
warning_fraction: float = 0.8 # Alert when at 80% of threshold
class SLOMonitor:
"""Monitors metrics against SLOs and generates alerts."""
def __init__(self, slos: List[SLOConfig]):
self.slos = slos
self.alerts: List[dict] = []
def check_slos(self, metrics: dict) -> List[dict]:
"""Check current metrics against SLOs."""
alerts = []
for slo in self.slos:
# Map SLO names to metric keys
metric_key = f"{slo.name}_p{int(slo.target_percentile * 100)}_ms"
if metric_key not in metrics:
continue
current_value = metrics[metric_key]
warning_threshold = slo.threshold_ms * slo.warning_fraction
if current_value > slo.threshold_ms:
alerts.append(
{
"severity": "necessary",
"slo": slo.name,
"message": f"{slo.name} P{int(slo.target_percentile * 100)} "
f"({current_value:.0f}ms) exceeds SLO ({slo.threshold_ms:.0f}ms)",
}
)
elif current_value > warning_threshold:
alerts.append(
{
"severity": "warning",
"slo": slo.name,
"message": f"{slo.name} P{int(slo.target_percentile * 100)} "
f"({current_value:.0f}ms) approaching SLO ({slo.threshold_ms:.0f}ms)",
}
)
return alerts# Define SLOs and check against current metrics
slos = [
SLOConfig("latency", target_percentile=0.95, threshold_ms=3000),
SLOConfig("latency", target_percentile=0.99, threshold_ms=5000),
SLOConfig("ttft", target_percentile=0.95, threshold_ms=500),
]
monitor = SLOMonitor(slos)
alerts = monitor.check_slos(metrics)SLO Status: -------------------------------------------------- [OK] latency P95: 940ms (target: 3000ms) [OK] latency P99: 1320ms (target: 5000ms) [OK] ttft P95: 259ms (target: 500ms) No active alerts
The monitor verifies that all performance metrics currently satisfy their service level objectives. By implementing both warning thresholds (at 80% of the SLO) and necessary thresholds (at 100%), the system provides early indications of degrading performance before actual SLA violations occur. Warning alerts give operators time to investigate and potentially scale up or address issues before users experience violations. Critical alerts indicate that the SLA has already been breached and immediate action is required.