Monitoring › Logging

While metrics summarize behavior in numbers, logs capture the raw events. For LLM services, logs are especially valuable because they preserve the actual inputs and outputs, enabling retrospective quality analysis and debugging. When an automated alert fires indicating quality degradation, your first response will be to look at the logs to understand what inputs triggered the problem.

A production LLM request log entry should capture enough information to fully reconstruct what happened during the request without needing to contact the user. The essential fields are:

• A unique request ID (for correlating across logs and traces)
• Timestamp and latency
• The model identifier and version
• Compressed or hashed input (preserving structure while protecting PII)
• Output text or a hash of the output
• Token counts (input and output separately)
• Any metadata relevant to routing (which backend handled the request, whether it was a cache hit)
• Quality scores from any automated evaluators
• User feedback if available

Logging the full input and output text is ideal for debugging but raises serious privacy and storage concerns. A practical middle ground is to log full text for a small random sample (say 1-5% of requests) while logging only metadata for the rest. This gives you enough data for quality analysis without storing everything. The sampling can be biased toward unusual requests: low quality scores, high latency, errors, and new user sessions are all good candidates for higher sample rates because they are more likely to contain useful debugging information.

Logs are most useful when they are structured rather than free-form text. A free-form log line like "Completed request abc123 in 1.2s with 423 tokens" requires regex parsing to extract values. A structured log in JSON format makes the same information immediately queryable:

{
  "request_id": "abc123",
  "latency_ms": 1200,
  "input_tokens": 156,
  "output_tokens": 267,
  "model": "gpt-4o",
  "quality_score": 0.87,
  "timestamp": "2024-01-15T10:23:45Z"
}

This structure enables log aggregation systems like Elasticsearch, Loki, or CloudWatch Logs Insights to run arbitrary queries without custom parsing logic. You can ask questions like "what is the average quality score for requests with more than 2,000 input tokens in the last seven days?" directly in the query interface, without writing a data pipeline first. Structured logging is one of those practices that seems like overhead until you need to debug a production incident under time pressure, at which point it becomes invaluable.

The field naming in structured logs deserves the same care as an API design. Choose names that are descriptive (input_tokens rather than itk), consistent across services (so joins are possible), and stable over time (changing a field name silently breaks all queries that relied on it). Many teams adopt a shared log schema across services to enable cross-service analysis.

Raw LLM logs can be enormous. A service handling 100,000 requests per day, each with a 1,000-token input and 500-token output, generates gigabytes of text per day. A practical retention strategy tiers logs by freshness and importance:

• Hot tier (0-7 days): Full logs including text, indexed for fast search. Used for active debugging.
• Warm tier (7-90 days): Metadata only (timestamps, token counts, quality scores), compressed. Used for trend analysis.
• Cold tier (90+ days): Aggregated statistics only, archived to cheap storage. Used for auditing and compliance.

The transition between tiers should be automated: your log management system drops text payloads at the warm tier boundary and drops individual records entirely at the cold tier boundary, retaining only rollup aggregations. This keeps storage costs manageable while preserving the data you need for each use case.