Caching › Caching in LLM Production Systems

Every call to a large language model carries a cost measured in dollars, milliseconds, and compute. When users ask the same question twice, or when a system prompt appears in every request, re-running the same computation from scratch wastes all three. Caching addresses this directly: store results you have already computed, and serve them instantly on the next matching request.

Caching is not a new idea. It sits at the foundation of computer architecture, operating systems, and web infrastructure. CPU caches store recently accessed memory words because programs tend to access the same addresses repeatedly. HTTP caches store web page responses because many users request the same pages. Content delivery networks push static assets to edge nodes because geographic proximity reduces latency. In every case, the same insight applies: computation or network round-trips are expensive, and reuse is cheap.

The difficult part of LLM caching is deciding when two requests should match. A CPU cache matches exact memory addresses. A web cache matches exact URLs. An LLM cache must decide what "the same" means for natural language queries, for long shared prefixes in tokenized prompts, and for structured API calls that include model parameters, sampling configuration, and multi-turn conversation history. Each level of the system offers different granularity, different savings, and different implementation tradeoffs.

The economics also favor caching. LLM inference is expensive relative to most computational tasks. Running a large model like GPT-4 or Claude can cost hundreds of times more than a typical database query or API call. Even a modest cache hit rate of 20-30% translates directly into a proportional reduction in compute spend. At scale, the same hit rate produces a substantial reduction in inference spend.

This chapter works through the main forms of LLM caching from the inside out. We begin with prompt caching, which operates at the model level and saves computation on repeated token prefixes. We then move to semantic caching, which operates at the application level and matches queries by meaning rather than exact string equality. We cover cache invalidation, the notoriously hard problem of knowing when a cached answer is no longer valid. Finally, we examine cache hit rates, the key metric for evaluating whether a caching strategy is helping.