Monitoring › Metrics

Metrics are numeric measurements sampled at regular intervals or computed per event. They are the fastest signal to collect and the easiest to alert on. For LLM services, metrics fall into several natural categories, each measuring a different dimension of system health.

Infrastructure metrics are the standard system-level signals that any service should track. They represent the mechanical health of the system independent of the model's output.

• Request rate: Requests per second arriving at the service. Sudden spikes indicate viral usage or potential abuse. Sudden drops indicate outages. The rate is also the foundation for understanding all other metrics: a latency increase is far more alarming at 500 requests per second than at 5 requests per second.

• Request latency: Time from request arrival to response completion, measured at p50, p95, and p99 percentiles. The p99 matters more than the mean because LLM latency distributions are heavily right-skewed: a small fraction of requests with very long outputs or high retry counts can have orders-of-magnitude higher latency than the median. If you only monitor the mean, that fraction is invisible. Users hitting those p99 requests experience the worst performance your system can produce.

• Error rate: Fraction of requests ending in an error. Segment by error type (model timeout, rate limit exceeded, output too long) to distinguish between infrastructure problems and usage patterns. An error rate spike caused by rate limiting from an upstream model provider requires a different response than one caused by your own inference backend running out of memory.

• Resource utilization: GPU memory and utilization, CPU, RAM, and network I/O. GPU memory deserves special attention because LLM inference is almost always memory-bound; even small memory leaks cause gradual degradation. A model loaded at half the GPU's memory capacity has room for batch growth; one that fills 90% of the GPU has very little margin before out-of-memory errors start occurring.

Beyond infrastructure, LLM services have operational metrics that do not exist in other software systems. These reflect the unique economics and mechanics of token-based language model inference.

Tokens per request is the number of input tokens and output tokens in each request. This is fundamental in a way that has no analogue in traditional services. A SQL query that returns a thousand rows costs roughly the same as one that returns ten rows, in terms of server-side compute. An LLM request with 8,000 input tokens costs far more to serve than one with 100 input tokens, both in time and money. Tracking token counts per request lets you understand your cost structure, identify unusually expensive requests, and set budget guardrails.

Tokens per second (throughput) measures the raw generation speed of the model: output tokens generated per second. This is the benchmark number that most hardware vendors advertise, and it is what ultimately determines the steady-state capacity of your inference backend. Tracking it continuously lets you detect performance regressions after software updates or configuration changes.

Time to first token (TTFT) is the latency from request arrival to the first token appearing in the response. For streaming applications, this is what users feel as "responsiveness," and it is distinct from end-to-end latency. A model that takes two seconds to produce its first token but then streams quickly may feel more responsive than one that takes one second to start but produces tokens slowly. TTFT is dominated by the prefill phase (processing the input prompt), so it scales with input length. Tracking TTFT separately from end-to-end latency lets you distinguish slow prefill from slow generation.

Cache hit rate measures the fraction of requests served from cache. If your system uses KV-cache or semantic caching, cache hits can reduce cost and latency by an order of magnitude. A dropping cache hit rate often signals that query patterns are shifting, which might also predict increases in cost and latency before those increases show up in other metrics.

Context length distribution tracks the distribution of prompt lengths over time. If users are feeding in longer and longer contexts (because they are pasting entire documents, or because conversation history is accumulating), your latency and cost will increase, and you need to know this proactively. Context length also affects memory pressure: longer contexts require more GPU memory for the KV cache during inference, and a sudden shift toward very long contexts can push a system that was comfortably within memory limits into out-of-memory territory.

Quality metrics are harder to compute automatically but are the most important for understanding whether your system is doing its job. An LLM service that is fast and reliable but produces bad answers is still a bad service. Quality metrics are what distinguish LLM monitoring from generic API monitoring.

Automated quality scores use secondary models or simpler classifiers to score each response for relevance, coherence, safety, or task-specific properties. For example, a retrieval-augmented generation (RAG) system might score each response for faithfulness to the retrieved documents using a small entailment model. These scores are inherently imperfect but provide continuous coverage that human review cannot match. The key is to calibrate automated scores against human judgments periodically to ensure the scorer remains aligned with the properties you care about.

User feedback signals capture real user judgment at scale. Thumbs up/down ratings, regeneration rates (users who click "try again"), edit rates (users who modify the model's output before using it), and session abandonment rates all proxy for response quality. These signals are noisy - a user might regenerate because of curiosity rather than dissatisfaction - but aggregated across thousands of requests they provide a reliable quality signal that is independent of any model-based evaluator.

Safety classifier outputs track the fraction of responses that trigger or nearly trigger safety filters. If your application has safety requirements, running each response through a classifier and tracking the rate over time lets you detect shifts in user behavior (more adversarial inputs arriving) or model behavior (safety degradation after a fine-tuning update).

The combination of automated scores, user feedback, and safety monitoring provides a three-way view of quality. Automated scores are sensitive but imperfect; user feedback is ground truth but slow and incomplete; safety monitoring catches the tail of unacceptable outputs. Together they give you much higher confidence that the system is performing well than any single signal alone.