Monitoring › The Monitoring Stack

A complete monitoring system for an LLM service consists of four layers, each building on the previous one.

Instrumentation is the code embedded in your application that generates raw signals: counters, timers, gauges, and log entries that describe what the system is doing at each moment. Good instrumentation is lightweight, consistent, and designed for the specific questions you need to answer. For LLM services, this means tracking the usual HTTP metrics along with token counts, quality scores, and model-specific signals like time to first token.

Collection and storage is the infrastructure that receives those signals and persists them efficiently. Metrics need a time-series database that handles high write throughput and supports range queries (give me the p99 latency for all requests over the last four hours). Logs need a system that can handle large text volume, compress it efficiently, and support full-text and structured queries. Traces need a system that can stitch together spans from multiple services into a coherent call graph.

Visualization is the dashboard layer that renders signals in human-readable form. A good dashboard does not just display numbers; it presents information in a way that makes patterns immediately apparent. Operators who stare at a dashboard for thirty seconds should know whether anything requires attention. This requires careful design choices: which metrics to show, how to normalize them, which time window to default to.

Alerting is the rules layer that automatically notifies operators when signals cross thresholds. Alerting closes the loop between monitoring and action. Without it, monitoring is purely reactive: operators look at dashboards when they suspect a problem. With good alerting, problems surface themselves before users notice them.

These layers are commonly implemented using the open-source LGTM stack (Loki for logs, Grafana for dashboards, Tempo for traces, and Mimir or Prometheus for metrics), though cloud-native alternatives like Datadog, New Relic, or CloudWatch are equally viable. The choice between self-hosted open source and managed cloud services involves tradeoffs in operational burden, cost, data residency, and integration depth. Teams running on Kubernetes often find the open-source stack integrates more naturally with their existing tooling, while teams at early stages often prefer a managed service to avoid the overhead of running yet more infrastructure.