Monitoring › Distributed Tracing

Metrics tell you that something is slow. Logs tell you what happened during a specific request. Traces tell you where time was spent across the entire call chain. For a typical LLM application, a single user request might involve several services: a web frontend, an API gateway, a vector database lookup, a prompt assembly service, the model inference backend, and a response post-processing step. Tracing attaches a unique trace ID to the request at entry and propagates it through every service, creating a timeline that shows exactly where latency is coming from.

For LLM services specifically, traces answer questions like: is the latency increase coming from the vector database lookup, the model inference step, or post-processing? Without tracing, you might assume the model is slow when in fact your embedding lookup is the bottleneck. This distinction matters enormously for triage: optimizing the wrong layer wastes engineering time while the real problem continues.

A trace for an LLM request typically consists of several spans. The root span covers the entire request duration from receipt to response. A child span covers the prompt assembly step, showing how long it takes to retrieve context documents and construct the final prompt. Another child span covers the model inference call, showing the time spent in the inference backend. If the model call is to an external API, this span shows the round-trip latency to that provider. Post-processing steps, output validation, and logging each get their own spans.

OpenTelemetry has become the standard instrumentation library for distributed tracing. It provides language-specific SDKs that instrument your code with minimal effort and export trace data to any compatible backend. The key advantage of OpenTelemetry is vendor neutrality: the same instrumentation code can export to Jaeger, Tempo, Zipkin, or any commercial tracing backend, so you are not locked into a specific vendor's SDK. As your monitoring infrastructure evolves, you can change the backend without re-instrumenting your application.