Caching › Summary

Caching in LLM production systems operates at multiple levels, each targeting a different source of repeated computation.

Prompt caching works at the model level by reusing pre-computed KV tensors for shared prompt prefixes. It is transparent to application code and can reduce time-to-first-token by 80-90% for requests with long shared system prompts. The key requirement is placing stable content at the beginning of the prompt and keeping it exactly consistent across requests. Providers enforce minimum prefix lengths (typically 1,024 tokens) and use time-based eviction with windows of 5 minutes to 1 hour.

Semantic caching works at the application level by matching queries on semantic similarity rather than exact text equality. It uses text embeddings and vector similarity search to find cached responses for queries that are paraphrases of previously answered questions. The similarity threshold is the critical hyperparameter, balancing hit rate against the risk of returning incorrect cached answers. Context-aware caching handles the multi-turn case by including conversation history in the matching key.

Cache invalidation is the hardest part of caching. Time-based TTL is simple but blunt. Event-based invalidation is precise but requires dependency tracking. Version keys provide a clean mechanism for flushing stale entries when shared state changes. Semantic caches require special handling for invalidation because entries cannot be identified by deterministic keys, often requiring side indices or full-flush strategies.

Cache hit rates are the primary metric for evaluating caching effectiveness. Workloads following a Zipfian distribution, as most natural language workloads do, deliver disproportionate hit rates from small caches. Hit rate, combined with latency reduction and cost savings, provides the full picture of caching value. Aggregate hit rate alone can be misleading; per-query-type and similarity-score distributions give a sharper view of cache health.

The practical approach for production systems is to layer these mechanisms: exact-match caching for identical requests, semantic caching for paraphrases, and prompt caching at the model level for all requests. Together, these layers cover the most common sources of redundant computation and deliver meaningful improvements to cost, latency, and throughput. Monitoring, threshold calibration, and cache warming are the ongoing operational work that keeps each layer performing well as traffic patterns and content evolve.