Caching › Limitations and Practical Implications

Caching is powerful but limited by the nature of LLM applications. The first and most persistent limitation is that not all tasks are cacheable. Creative generation, personalized content, and queries requiring fresh real-world information resist caching because no two requests have the same correct answer. Applying semantic caching to these tasks will produce responses that seem plausible but are contextually wrong, which can be more harmful than simply running inference. Before instrumenting a caching layer, you should audit your traffic to understand what fraction of requests are cacheable.

Semantic caching introduces a subtle failure mode: the embedding model and the generation model may not agree on what "similar" means. Two queries can be close in embedding space but require different responses. For example, "What is the boiling point of water at sea level?" and "What is the boiling point of water at high altitude?" are semantically similar (both about water's boiling point) but have different answers (100 degrees Celsius vs. roughly 90 degrees Celsius). An embedding model trained for general semantic similarity may assign these queries a high cosine similarity, causing the cache to return the wrong answer. Domain-specific embedding models and higher thresholds reduce but do not eliminate this risk. For safety-critical or high-stakes applications, you should validate the cache against a labeled dataset of known-similar and known-distinct query pairs before deploying it.

Prompt caching has a different set of limitations. It requires that the shared prefix be truly identical at the token level. Systems that inject dynamic content into system prompts, such as timestamps, request IDs, or per-user personalization strings, lose prompt caching benefits unless the dynamic content is carefully isolated at the end of the prompt. The discipline of ordering prompt components from most stable to least stable is not always compatible with other design goals, particularly in highly personalized systems where the personalization is conceptually part of the "setup" that comes before the user's question.

Cache invalidation remains an open engineering problem. The strategies described in this chapter, TTL, event-based invalidation, and version keys, all involve tradeoffs between freshness, complexity, and performance. Systems that serve frequently updated information must choose between serving stale cached responses and paying the full cost of inference on every request. There is no free lunch: the only way to guarantee freshness is to not cache, which means giving up the performance and cost benefits entirely. Good production systems typically accept a defined maximum staleness, communicate it to users where relevant, and design their cache TTLs to keep staleness within that bound.

A practical concern that often goes unaddressed in initial caching implementations is the effect of caching on model output diversity. Many LLM applications benefit from some variability in responses: slightly different wordings feel more natural to users and reduce the sensation of talking to a machine. Aggressive caching reduces this variability. If the same response is returned for every paraphrase of "what is the weather like?", conversations start to feel repetitive. For applications where conversational naturalness matters, you can introduce a randomization layer that occasionally bypasses the cache even for high-similarity matches, preserving some response diversity at the cost of a slightly lower effective hit rate.

Scalability is a final concern. An in-process Python cache of a few thousand entries (as implemented above) works for low-traffic prototypes. A production system with millions of cached entries distributed across many replicas requires a different architecture: a centralized vector database, a distributed key-value store for responses, and careful cache key partitioning to avoid hot spots. The embedding computation itself scales to dedicated inference endpoints if the volume requires it. As caching infrastructure becomes more complex, the operational overhead of maintaining it must be weighed against the cost savings it delivers.

Despite these limitations, caching consistently delivers some of the best return on investment of any latency optimization in LLM systems. In systems where even 10-20% of traffic can be served from cache, the cost and latency reductions are significant enough to change the economics of deployment. The engineering effort required to implement a basic exact-match or semantic cache is modest compared to the gains, making caching one of the first optimizations to add when moving from prototype to production.