Monitoring › Dashboards

A monitoring dashboard should answer three questions at a glance: Is the system up? Is it performing well? Is the output quality acceptable? Everything else is detail that should be available on drill-down.

A practical LLM service dashboard organizes panels into three rows. The top row shows the health signals that require immediate attention: request rate, error rate, and p99 latency. If any of these is anomalous, an operator knows immediately that something requires investigation. The second row shows resource utilization: GPU memory and compute, queue depths, and active connection counts. These inform capacity planning and root cause analysis. The third row shows quality signals: rolling quality scores, safety classifier hit rates, and user feedback metrics. These are slower-moving signals but often the first to reveal model drift.

Dashboard design benefits from two principles borrowed from industrial process control. The first is "management by exception": normal is not interesting. Configure charts to show the deviation from baseline rather than the absolute value. A latency chart that shows "300 ms above normal" is more actionable than one showing "1,500 ms absolute," because the reader does not need to remember what "normal" looks like to understand that something is wrong. The second principle is information hierarchy: aggregate at the top, disaggregate on demand. Start with a single number for each metric and let operators click through to the distribution or the individual request trace. Dashboard clutter is a form of alert fatigue: if operators have to scan 40 panels to assess system health, they will either miss problems or stop looking at the dashboard altogether.

The time range selector is one of the most important controls on a dashboard. Short windows (last five minutes) reveal transient spikes and active incidents. Medium windows (last 24 hours) show intraday patterns like traffic peaks during business hours. Long windows (last 30 days) reveal trends: gradual latency increases, slowly declining quality scores, or growing token usage that predict future capacity problems. A good dashboard defaults to the last one or two hours for operational monitoring but makes it easy to zoom out for trend analysis.

Grafana's built-in alert annotations are one of its most useful features for LLM monitoring. When an alert fires, Grafana can mark the corresponding time on every panel in the dashboard with a colored annotation line. This makes it trivial to correlate an alert with the underlying metrics: you can see immediately whether the error rate spike preceded the quality score drop or followed it, which often points toward the root cause.