Monitoring › Limitations and Practical Challenges

The most significant limitation of LLM monitoring is the fundamental difficulty of automatically measuring output quality. Almost every metric discussed in this chapter is either a proxy for quality (latency, error rate) or requires a secondary model to compute (automated quality scores). Secondary models bring their own accuracy limitations: a quality scoring model that gives high scores to plausible-sounding but wrong answers provides false assurance. The only ground truth for output quality is human judgment, which does not scale to production traffic volumes.

Practical teams address this through sampling strategies combined with calibration loops. Rather than trying to evaluate every request, they maintain a human review queue that samples a small fraction of production traffic, with intelligent sampling that over-represents edge cases, low-confidence predictions, and new input types. Human-reviewed samples serve two purposes: they provide direct quality signal for the sampled requests, and they feed back into training data for the automated quality scorer. Over time, the scorer becomes better calibrated to the specific quality dimensions that matter for that application.

Privacy is another real constraint. Logging full request and response text exposes potentially sensitive user data. Organizations subject to GDPR, HIPAA, or similar regulations need to think carefully about what they can log, how long they can retain it, and who can access it. In practice, many teams log only metadata and hashes for the vast majority of requests, with full text logging restricted to consented opt-in test users or synthetic requests. This restriction limits the depth of quality analysis possible from logs, which is a real operational cost of privacy compliance. Some teams handle this by having users explicitly opt into full logging in exchange for a quality guarantee or other benefit.

Monitoring infrastructure itself can fail. If your Prometheus scraper falls behind, you lose metric data. If your Loki ingest pipeline is overloaded, log entries are dropped. If your alerting system has a bug, alerts do not fire. Production monitoring requires its own monitoring: track the health of the collection and storage layer, alert on ingest lag, and verify alerts by periodically injecting known anomalies into the system (a practice sometimes called "chaos testing for observability"). An unmonitored monitoring system is worse than no monitoring system, because it gives you false confidence.

Finally, monitoring adds overhead. Each instrumentation call, each metric update, and each log write consumes CPU cycles and I/O. For high-throughput services, the monitoring infrastructure can become a bottleneck. The standard mitigation is to do most monitoring work asynchronously: record metrics to a local in-process buffer and flush to the collector in a background thread, rather than blocking the request path for each metric update. Log writes should go to a local buffer that is flushed periodically or when it fills, not synchronously on each request. Tracing spans should be sampled at the trace level (keep 1% of traces end-to-end) rather than the span level, to avoid the overhead of recording every span.