Caching › Caching Architecture in Practice

Understanding individual caching mechanisms is necessary but not sufficient. A production system typically operates multiple caching layers simultaneously, and the interaction between them determines overall performance.

A well-designed LLM application cache has at least three layers, each targeting a different source of efficiency.

The first layer is exact-match caching at the request level. Before doing anything else, hash the complete request (including the full prompt text, model parameters, and any other request fields) and check a fast key-value store like Redis. If the hash matches a cached response, return it immediately. This handles repeated identical requests at near-zero latency. It is deterministic and works well when users or automated pipelines make repeated identical requests. API clients that retry failed requests, scheduled jobs that re-run with the same inputs, and automated test suites that repeat the same queries all benefit disproportionately from this layer.

The second layer is semantic caching for queries that are equivalent in meaning but not textually identical. This runs only if the exact-match cache misses. It uses the embedding and vector search approach described earlier. The latency is higher than exact matching (typically 5-20 ms for the embedding lookup plus vector search) but still much lower than a full model inference call. The semantic cache catches the paraphrase traffic that the exact-match cache misses.

The third layer is prompt caching at the model inference level. This runs automatically if the LLM provider supports it and requires no application-layer logic. It benefits all requests that share a long system prompt or common context prefix, regardless of whether they hit the application-level cache. Every request to Claude, GPT-4, or Gemini benefits from prompt caching whenever the system prompt is shared and meets the minimum length threshold.

The combination means that the most common requests (exact repeats) are handled instantly, semantically equivalent requests are handled quickly, and all requests benefit from reduced time-to-first-token through prompt caching. Each layer handles what the previous layer cannot.

The exact-match cache key must uniquely identify a request while being stable for requests that should match. Poor key design is a common source of cache miss inflation.

A naive approach is to hash the entire HTTP request body, including all parameters. This fails if parameters are serialized in a different order on each request, or if non-semantic fields like request IDs or timestamps are included. Two requests that are semantically identical but carry different request IDs will produce different hashes and both miss the cache.

A reliable cache key includes:

• The model identifier and version
• The full prompt text (normalized to a canonical form)
• Generation parameters that affect the output (temperature, max tokens, stop sequences)
• Any sampling seed if deterministic outputs are required

Fields that should not be included: request timestamps, request IDs, tracing headers, and any session or authentication metadata that does not affect the model's output.

For responses that vary by user (personalized outputs), you must include a user identifier in the cache key. This reduces the hit rate but ensures users do not receive each other's cached responses.

A cache with no entries provides no benefit. Cache warming is the practice of pre-populating a cache before traffic arrives, using knowledge of what requests are likely.

For exact-match and semantic caches, warming means making representative requests, or simulating them, during deployment so that common queries already have cached responses when real traffic arrives. For prompt caching, warming means sending requests that include the system prompt before the cache is needed, so the KV state is already stored.

Cache warming is particularly important for systems with long cache eviction windows or for deployments that restart frequently. A cache that gets cold between deploys provides inconsistent latency, which is harder to reason about than uniformly slow uncached responses. Users who happen to arrive during a cold cache period experience full inference latency; users who arrive later experience cache-accelerated latency. This inconsistency can make performance analysis difficult.

Effective warming strategies include replaying the previous day's or hour's most common requests during deployment, pre-computing responses for the top-K queries identified from query logs, and using synthetic query generation to cover the most probable queries for a given domain. The cost of warming is paid once per deployment, and the benefit accrues for the lifetime of the cache.

Caching without monitoring is incomplete. You need to know whether the cache is working as intended, whether quality is being maintained, and whether the cache is cost-effective to operate.

Key metrics to track include:

• Hit rate broken down by cache layer (exact match, semantic, prompt)
• Similarity score distribution for semantic cache hits
• Latency percentiles (p50, p95, p99) for cached and uncached requests
• Cache storage utilization and eviction rates
• Cost per request with and without caching

The similarity score distribution is particularly important for semantic caches. If most hits are occurring at similarities of 0.85-0.88 and your threshold is 0.88, you are close to the quality boundary. Drifting the threshold down without reviewing hit quality can silently degrade response accuracy.

User feedback signals, such as thumbs-up/thumbs-down ratings or downstream task performance, should be correlated with cache hit/miss status. If users rate cached responses lower than fresh responses, your threshold is likely too low or your cache domain is too broad.