Caching for LLMs: Prompt, Semantic, and Invalidation

Every call to a large language model carries a cost measured in dollars, milliseconds, and compute. When users ask the same question twice, or when a system prompt appears in every request, re-running the same computation from scratch wastes all three. Caching addresses this directly: store results you have already computed, and serve them instantly on the next matching request.

Caching is not a new idea. It sits at the foundation of computer architecture, operating systems, and web infrastructure. CPU caches store recently accessed memory words because programs tend to access the same addresses repeatedly. HTTP caches store web page responses because many users request the same pages. Content delivery networks push static assets to edge nodes because geographic proximity reduces latency. In every case, the same insight applies: computation or network round-trips are expensive, and reuse is cheap.

The difficult part of LLM caching is deciding when two requests should match. A CPU cache matches exact memory addresses. A web cache matches exact URLs. An LLM cache must decide what "the same" means for natural language queries, for long shared prefixes in tokenized prompts, and for structured API calls that include model parameters, sampling configuration, and multi-turn conversation history. Each level of the system offers different granularity, different savings, and different implementation tradeoffs.

The economics also favor caching. LLM inference is expensive relative to most computational tasks. Running a large model like GPT-4 or Claude can cost hundreds of times more than a typical database query or API call. Even a modest cache hit rate of 20-30% translates directly into a proportional reduction in compute spend. At scale, the same hit rate produces a substantial reduction in inference spend.

This chapter works through the main forms of LLM caching from the inside out. We begin with prompt caching, which operates at the model level and saves computation on repeated token prefixes. We then move to semantic caching, which operates at the application level and matches queries by meaning rather than exact string equality. We cover cache invalidation, the notoriously hard problem of knowing when a cached answer is no longer valid. Finally, we examine cache hit rates, the key metric for evaluating whether a caching strategy is helping.

Prompt caching exploits the internal mechanics of transformer inference. To understand why it saves computation, you need to understand what the model does when it processes a prompt.

When a transformer processes a sequence of tokens, each attention layer computes key and value matrices for every token in the input. These matrices encode each token's representation in the context of every other token it can attend to. Computing them requires matrix multiplications that scale with both the sequence length and the model's hidden dimension, making them the dominant cost in long-context inference.

The key and value matrices for a given token depend only on that token's position and all preceding tokens. They do not depend on what comes after. This property is what makes prefix reuse possible.

The essential observation is that if two prompts share a prefix, the key and value matrices for that prefix are identical in both requests. There is no mathematical reason to recompute them. Prompt caching implements exactly this optimization: the computed KV tensors for a prefix are stored in fast memory and reused whenever a new request shares that prefix, bypassing the most expensive portion of the forward pass.

The cost savings are substantial. A request whose prompt is 90% shared prefix and 10% new content requires the model to compute full attention only over the new tokens, dramatically reducing time-to-first-token (TTFT). For requests where the system prompt is 2,000 tokens and the user message is 50 tokens, prompt caching can reduce TTFT by 80-90%. The financial savings are equally large because most providers charge for input tokens processed, and cached tokens are processed at a significant discount or not charged at all.

Not all prompts can benefit equally from prefix caching. The benefit depends on how prompts are structured relative to the caching mechanism.

Prefix caching matches on the exact sequence of token IDs from the beginning of the prompt. If a shared component appears anywhere other than the beginning, it does not help. This is why prompt engineering for production systems often includes a discipline: place stable, reusable content at the front. A typical ordering runs from most stable to least stable:

• System prompt (instructions, persona, output format)
• Retrieved documents or context (may vary per query but often repeated across sessions)
• Few-shot examples
• The user's actual question

This ordering ensures that as many leading tokens as possible are shared across requests, maximizing the prefix that can be cached. Violating this ordering, for example by placing user-specific metadata at the beginning, can eliminate all caching benefit even when 90% of the prompt content is shared.

The granularity of caching is typically at the cache block level, where each block covers a fixed number of tokens (commonly 16 or 32). A prefix must be long enough to fill at least one complete block to be eligible for caching. Short system prompts of only a few tokens may not save as much as expected because the partial final block must still be recomputed. In practice, this means that very short system prompts gain little from prefix caching even when they are consistently shared.

Most providers require a minimum prompt length before activating prompt caching because the overhead of cache lookup and storage for very short prompts exceeds the savings. The cache lookup itself requires hashing the prefix, searching the cache index, and loading stored tensors from memory, all of which take time. For very short prefixes, this overhead may exceed the savings from not recomputing the prefix.

Anthropic's Claude API activates prompt caching at 1,024 tokens for Claude 3 models. OpenAI's context caching for GPT-4o activates at 1,024 tokens as well. Google's Gemini API uses a minimum of 32,768 tokens for its explicit caching feature, targeting use cases with very long shared context such as entire code repositories or long documents. These thresholds reflect each provider's judgment about where the overhead-to-savings ratio becomes favorable.

Understanding minimum lengths matters for system design. A system prompt that is 800 tokens long sits below the caching threshold and will be fully recomputed on every request regardless of how consistently it is shared. Padding to reach the threshold by adding more thorough instructions is sometimes worth doing.

Cached KV states are not stored indefinitely. They consume GPU memory, the scarcest resource in LLM deployments. A single cached context for a 2,000-token prompt in a large model can occupy tens of megabytes of GPU memory. Multiply this across thousands of distinct prefixes and the memory pressure becomes significant.

Providers implement time-based eviction: a cached entry is retained for a sliding window (typically 5 minutes to 1 hour) and evicted if no new request uses it within that window. The eviction policy resembles a least-recently-used (LRU) scheme at the time dimension: entries that have not been accessed recently are removed to free memory for new entries. Systems with bursty but infrequent traffic may see poor cache hit rates because the cached state is evicted between bursts.

This eviction behavior has practical implications for how you think about caching economics. A batch job that runs hourly and shares a long system prompt will likely miss the cache on every run unless you either increase the cache lifetime or keep the cache warm with periodic requests. Designing around eviction windows requires knowing your traffic pattern.

Some providers expose cache lifetime controls that allow you to trade memory usage for higher hit rates. A longer cache lifetime means higher hit rates for infrequent-but-repeated workloads at the cost of holding more GPU memory across longer periods. The right setting depends on your traffic burstiness and the ratio of cache storage cost to inference cost.

The standard prompt caching described above requires exact prefix matching at the token level. Two prompts share a cached prefix only if their token sequences are literally identical from token 0 through some boundary. A single changed character, even in whitespace, invalidates the match.

This strictness is intentional and mathematically necessary. The KV cache stores tensors that encode exact positional relationships. The key and value for a token at position $i$ in the attention computation depends on the token's embedding, the positional encoding for position $i$, and the learned weight matrices. Reusing a cached KV state for a slightly different prefix would produce incorrect outputs because the attention patterns would be based on wrong positional information.

The practical implication is that prompt templates must be constructed with care. Dynamic content such as timestamps, user IDs, session identifiers, or request-specific metadata should go at the end of the prompt, after all shared content. Even a date injected into the system prompt ("Today's date is March 15, 2025") will prevent caching if it changes between requests, because the token "March" at position 10 will have a different KV representation than "April" at the same position.

Systems that inject per-user personalization into the system prompt must decide whether to sacrifice caching for personalization or restructure their prompts to isolate the personalizing content at the tail. Often, the best solution is to move lightweight personalization information (name, preferences) into the final human turn rather than the system prompt, preserving the shared system prompt prefix.

Prompt caching benefits are visible in two metrics: TTFT and cost. TTFT is measured as the time from request submission to the first token of the model's response. Cost is measured in dollars per request.

For a prompt with $P$ prefix tokens and $N$ new tokens, where the prefix is fully cached, the effective input token count for billing is $N$ rather than $P + N$. If the cached token discount is $d$ (e.g., 0.10 for a 90% discount on cached tokens), the cost is:

where $c_{\text{input}}$ is the per-token input price and $d$ is the cached token fraction (near 0 for fully cached prefixes).

For TTFT, the savings come from skipping the forward pass over the prefix. The reduction is roughly proportional to the fraction of the prompt that is cached, because the forward pass time scales linearly with the number of tokens processed.

Prompt caching operates within the model inference pipeline and is invisible to application code. Semantic caching operates at the application layer and requires explicit design decisions about what to cache, how to measure similarity, and when to return a cached answer versus generating a fresh one.

The core idea is simple: if a user asks "What is the capital of France?" and a later user asks "Which city is France's capital?", these questions have the same intent and the same answer. Why run a full LLM inference for the second request when you already have the answer? Exact-match caching would miss this case because the two strings differ. Semantic caching catches it by comparing meaning rather than text.

This capability matters because natural language is inherently paraphrastic. People express the same idea in hundreds of different ways. Customer support databases receive thousands of slight variations of the same common questions. Knowledge base assistants handle paraphrases of the same factual lookups. Without semantic caching, each variation requires a fresh model call even when the answer is already known.

The standard architecture for semantic caching uses text embeddings to measure query similarity. The intuition is that a well-trained embedding model places semantically similar texts close together in a high-dimensional vector space, regardless of surface-level phrasing differences.

When a new query arrives, you compute its embedding using an embedding model, search a vector store for the nearest cached query embedding, and return the cached response if the similarity exceeds a threshold.

The process works as follows. For each query that generates a fresh response, you store the query embedding and the response together in the cache. On subsequent requests, you compute the embedding for the new query and run a nearest-neighbor search over stored embeddings. If the closest match has a cosine similarity above your threshold, you return the cached response. Otherwise, you run inference and cache the new result.

This architecture separates the matching problem from the storage problem. The vector store handles similarity search efficiently, using approximate nearest-neighbor algorithms that can search millions of vectors in milliseconds. The cache store handles response retrieval, using a key-value store indexed by embedding identifier.

The choice of embedding model matters significantly. You want embeddings that capture semantic meaning precisely enough to distinguish questions with different answers, while being similar enough for paraphrases of the same question. Models like text-embedding-3-small, all-MiniLM-L6-v2, and specialized task-specific embeddings are common choices. The embedding model adds latency to each cache lookup, so small, fast models are preferred for latency-sensitive applications. A model that takes 100 ms to compute an embedding makes semantic caching impractical if your target uncached latency is 150 ms.

The similarity threshold is the most consequential hyperparameter in semantic caching. It controls the precision-recall tradeoff between cache hits and response quality.

A high threshold (say, 0.95) means only near-identical paraphrases match. You get very few false cache hits where semantically different questions receive the wrong cached answer. But the cache hit rate will be lower because slightly different phrasings do not match.

A low threshold (say, 0.70) means more queries match, increasing the hit rate. But you risk returning incorrect cached answers for queries that seemed similar but had different intent. A question about "the effects of caffeine on sleep" might incorrectly match "the effects of melatonin on sleep" at a low threshold, returning an answer about the wrong substance.

There is no universally correct threshold. The right value depends on the domain, the diversity of your query space, and the cost of a wrong answer. Factual question-answering over a narrow domain can tolerate lower thresholds because similar questions have similar answers. Creative or personalized tasks should use higher thresholds or disable semantic caching entirely.

The most rigorous way to set a threshold is empirical. Collect a labeled dataset of query pairs with human judgments of whether they have the same answer. Compute embedding similarities for all pairs. Find the similarity value that maximizes the F1 score between hit rate (recall) and precision (fraction of hits that are correct). This calibration should be redone whenever the embedding model or the knowledge domain changes significantly.

Basic semantic caching stores individual queries and their responses. A more structured approach is cluster-level caching, where you identify clusters of semantically similar queries and cache a canonical response for the cluster. New queries are matched to the nearest cluster rather than to individual cached queries.

Cluster-level caching has better space efficiency: instead of storing one response per unique query, you store one response per semantic cluster. It also provides more predictable behavior because the response for each cluster is explicitly defined rather than depending on which query happened to arrive first. If the first query in a cluster was phrased unusually and received an unusually phrased response, subsequent queries in the cluster will receive that unusual response. With cluster-level caching, the canonical response is deliberately crafted or quality-reviewed.

The tradeoff is that cluster-level caching requires upfront work to define clusters and their canonical responses. It works well for structured domains like customer support, where query types are well-defined and the set of clusters is relatively stable over time. It is harder to maintain for open-domain systems where new query types emerge continuously.

A hybrid approach that works in many production systems is to run cluster-level caching for the high-frequency query types that can be identified in advance, and fall back to individual query-level caching for the long tail. The top 50 customer support question types might be cluster-cached with curated answers, while the remaining diversity of queries uses standard semantic caching.

Semantic caching treats queries as independent. In a conversational system, this assumption breaks down. The meaning and appropriate response to "What do you mean?" depends entirely on the preceding conversation. "Tell me more about that" is meaningless without context. "Is it expensive?" requires knowing what "it" refers to.

A semantic cache that matches these queries to cached responses from different conversations will almost certainly return something irrelevant or misleading. The embedding for "Can you clarify?" is similar in every conversation, but the correct response is completely different in each case.

Context-aware caching addresses this by including conversation history in the cache key. The simplest approach is to embed the last few turns of conversation alongside the current query and match on the concatenated representation. This prevents false matches across different conversations at the cost of a lower cache hit rate, since the combined representation is more unique.

The depth of context to include involves a tradeoff. Including only the last turn is cheap and provides some context disambiguation. Including the last five turns provides much better disambiguation but makes the embedding more expensive to compute and the cached representation less reusable (since the probability of two sessions having identical recent history is very low).

An alternative approach that avoids this tradeoff is to restrict semantic caching to the first turn of a conversation, where there is no context to contaminate the match. The first user message in a conversation is most likely to be a direct, context-free question that matches well across sessions. Subsequent turns within a conversation are handled by exact-match caching keyed on the full conversation history, or by skipping caching entirely for multi-turn context.

At small scale, you can implement semantic caching with an in-memory list of embeddings searched by brute-force cosine similarity. This works for caches with a few thousand entries and single-server deployments.

At production scale, you need a vector database that supports efficient approximate nearest-neighbor search. Options include Pinecone, Weaviate, Qdrant, and Milvus for dedicated vector databases, as well as PostgreSQL with pgvector or Redis with RedisSearch for embedding support in existing infrastructure. The right choice depends on your existing stack, your query volume, and your tolerance for managing additional infrastructure.

The latency of the vector search must fit within your overall latency budget. A vector database query typically completes in 5-20 ms for collections of millions of vectors using optimized index structures like HNSW (Hierarchical Navigable Small World graphs). Adding the embedding computation time (5-30 ms depending on model size), the total cache lookup overhead is typically 10-50 ms. This is well worth the trade for requests that would otherwise take 1-10 seconds for inference.

Understanding individual caching mechanisms is necessary but not sufficient. A production system typically operates multiple caching layers simultaneously, and the interaction between them determines overall performance.

A well-designed LLM application cache has at least three layers, each targeting a different source of efficiency.

The first layer is exact-match caching at the request level. Before doing anything else, hash the complete request (including the full prompt text, model parameters, and any other request fields) and check a fast key-value store like Redis. If the hash matches a cached response, return it immediately. This handles repeated identical requests at near-zero latency. It is deterministic and works well when users or automated pipelines make repeated identical requests. API clients that retry failed requests, scheduled jobs that re-run with the same inputs, and automated test suites that repeat the same queries all benefit disproportionately from this layer.

The second layer is semantic caching for queries that are equivalent in meaning but not textually identical. This runs only if the exact-match cache misses. It uses the embedding and vector search approach described earlier. The latency is higher than exact matching (typically 5-20 ms for the embedding lookup plus vector search) but still much lower than a full model inference call. The semantic cache catches the paraphrase traffic that the exact-match cache misses.

The third layer is prompt caching at the model inference level. This runs automatically if the LLM provider supports it and requires no application-layer logic. It benefits all requests that share a long system prompt or common context prefix, regardless of whether they hit the application-level cache. Every request to Claude, GPT-4, or Gemini benefits from prompt caching whenever the system prompt is shared and meets the minimum length threshold.

The combination means that the most common requests (exact repeats) are handled instantly, semantically equivalent requests are handled quickly, and all requests benefit from reduced time-to-first-token through prompt caching. Each layer handles what the previous layer cannot.

The exact-match cache key must uniquely identify a request while being stable for requests that should match. Poor key design is a common source of cache miss inflation.

A naive approach is to hash the entire HTTP request body, including all parameters. This fails if parameters are serialized in a different order on each request, or if non-semantic fields like request IDs or timestamps are included. Two requests that are semantically identical but carry different request IDs will produce different hashes and both miss the cache.

A reliable cache key includes:

• The model identifier and version
• The full prompt text (normalized to a canonical form)
• Generation parameters that affect the output (temperature, max tokens, stop sequences)
• Any sampling seed if deterministic outputs are required

Fields that should not be included: request timestamps, request IDs, tracing headers, and any session or authentication metadata that does not affect the model's output.

For responses that vary by user (personalized outputs), you must include a user identifier in the cache key. This reduces the hit rate but ensures users do not receive each other's cached responses.

A cache with no entries provides no benefit. Cache warming is the practice of pre-populating a cache before traffic arrives, using knowledge of what requests are likely.

For exact-match and semantic caches, warming means making representative requests, or simulating them, during deployment so that common queries already have cached responses when real traffic arrives. For prompt caching, warming means sending requests that include the system prompt before the cache is needed, so the KV state is already stored.

Cache warming is particularly important for systems with long cache eviction windows or for deployments that restart frequently. A cache that gets cold between deploys provides inconsistent latency, which is harder to reason about than uniformly slow uncached responses. Users who happen to arrive during a cold cache period experience full inference latency; users who arrive later experience cache-accelerated latency. This inconsistency can make performance analysis difficult.

Effective warming strategies include replaying the previous day's or hour's most common requests during deployment, pre-computing responses for the top-K queries identified from query logs, and using synthetic query generation to cover the most probable queries for a given domain. The cost of warming is paid once per deployment, and the benefit accrues for the lifetime of the cache.

Caching without monitoring is incomplete. You need to know whether the cache is working as intended, whether quality is being maintained, and whether the cache is cost-effective to operate.

Key metrics to track include:

• Hit rate broken down by cache layer (exact match, semantic, prompt)
• Similarity score distribution for semantic cache hits
• Latency percentiles (p50, p95, p99) for cached and uncached requests
• Cache storage utilization and eviction rates
• Cost per request with and without caching

The similarity score distribution is particularly important for semantic caches. If most hits are occurring at similarities of 0.85-0.88 and your threshold is 0.88, you are close to the quality boundary. Drifting the threshold down without reviewing hit quality can silently degrade response accuracy.

User feedback signals, such as thumbs-up/thumbs-down ratings or downstream task performance, should be correlated with cache hit/miss status. If users rate cached responses lower than fresh responses, your threshold is likely too low or your cache domain is too broad.

Cache invalidation is famously described as one of the two hard problems in computer science, alongside naming things. In LLM systems, it is difficult because the relationship between a cached response and its continued validity is complex and often implicit.

A cached response was correct, or at least appropriate, when it was generated. It can become wrong for many reasons, and those reasons are not always visible from the cached response itself.

The underlying facts changed. If you cache the response "The current CEO of Company X is Alice Smith" and Alice Smith is replaced, the cached response is now factually wrong. External knowledge changes continuously and unpredictably. You cannot know in advance which cached responses depend on which facts.

The model changed. LLM providers update models regularly, sometimes changing behavior, safety policies, or output formatting conventions. A response cached under model version A may not be the response you would want from model version B. Model updates can change tone, verbosity, reasoning style, and safety thresholds in ways that make old cached responses look inconsistent with current behavior.

The context changed. A cached response that was appropriate for a certain user, time, or system state may not be appropriate when those conditions change. A system prompt updated to reflect a new product version should invalidate all cached responses that were generated under the old system prompt, because the instructions may now be different.

The user state changed. In personalized systems, a cached response tied to a user's preferences may become stale as the user's preferences evolve. A financial advisor bot that cached recommendations based on a user's risk tolerance should invalidate those recommendations if the user updates their risk profile.

These sources of staleness are fundamentally different from each other and require different invalidation strategies. There is no single mechanism that handles all of them correctly.

The simplest invalidation strategy is time-to-live (TTL): cached entries expire after a fixed duration. A TTL of 1 hour means no cached response is more than 1 hour old. This is easy to implement, requires no dependency tracking, and provides a clear guarantee to users about maximum response staleness.

The weakness of TTL-based invalidation is its bluntness. Stable information like mathematical definitions, historical facts, or fixed product descriptions could be cached indefinitely without staleness. Fast-changing information like stock prices, weather, or live event data should not be cached at all, or only for seconds. A fixed TTL either over-invalidates stable content (wasting cache capacity and adding unnecessary latency) or under-invalidates dynamic content (serving stale answers).

Tiered TTLs address this by assigning different expiration times to different content types. A system might use a 24-hour TTL for factual knowledge base responses, a 1-hour TTL for product information, and a 5-minute TTL for pricing or availability data. Implementing tiered TTLs requires your caching logic to know something about the content type of each response, either by tagging responses at generation time or by classifying them based on the query type.

The right TTL values are empirical. You should measure how quickly information in your domain changes, then set TTLs to be some fraction of that typical change interval. Setting a TTL based on intuition without measurement often results in either overly conservative (frequent re-computation) or overly aggressive (stale responses) caching behavior.

Event-based invalidation ties cache expiration to specific events rather than elapsed time. When the system prompt changes, all cached responses generated under the old system prompt are immediately invalidated. When a knowledge base is updated, all cached responses that might have drawn on the changed content are invalidated.

Event-based invalidation is more precise than TTL and avoids serving stale responses during the window between a change event and the next TTL expiration. If you deploy an updated system prompt at noon and your TTL is 1 hour, you could serve stale responses for up to an hour. With event-based invalidation, the old responses are invalidated at noon immediately.

But it requires the caching system to track dependencies: which cached responses were generated under which system prompt version, using which knowledge base version, and so on. This dependency graph can become complex. A response might have been generated under system prompt version 3, using knowledge base documents 17, 42, and 91. Updating document 42 should invalidate that cached response, but tracking this at response granularity requires recording which documents each response depended on at generation time.

Dependency tracking adds implementation complexity that many teams find difficult to justify. One practical simplification is cache versioning: include a cache version identifier in every cache key. When the system prompt or other shared state changes, increment the version. All previous cache entries automatically become unreachable because their keys no longer match, achieving a complete cache flush for the changed component without needing to track individual dependencies.

Version-key invalidation is essentially a full flush for the affected component. Every entry generated under the old version becomes unreachable at once, and the cache must rebuild from scratch for that component. This is clean and correct, but it sacrifices the warmth of all entries that were not affected by the change. If only 5% of your responses depended on the changed document, version flushing invalidates the other 95% unnecessarily.

When a change occurs, you have two options: invalidate only the entries likely affected by the change, or flush the entire cache and rebuild from scratch.

Full flush is simple and guaranteed correct. It is appropriate when the change is pervasive (such as a major model upgrade or a system-wide policy update) or when you cannot reason about which entries are affected. The cost is a cold cache immediately after the flush, with degraded hit rates until the cache warms up again. The warmup period can take hours or days depending on traffic volume, and during this period users experience full inference latency.

Partial invalidation is more efficient but requires careful reasoning about scope. If a specific FAQ article is updated, you might invalidate only the cached responses to questions about that article. This preserves cache warmth for unaffected queries. But if your invalidation logic is wrong and you miss some affected entries, you will serve stale responses for those queries, which is the failure mode you were trying to avoid by caching selectively.

The choice between full flush and partial invalidation should be governed by the expected scope of changes and your tolerance for serving stale responses. For infrequent, broad changes (model upgrades), full flush is safer. For frequent, narrow changes (individual document updates), partial invalidation pays off.

Semantic caches present an additional invalidation challenge that does not exist in exact-match caches. In an exact-match cache, you can invalidate a specific entry by its key. In a semantic cache, entries are matched by embedding similarity, not by a deterministic key. You cannot easily invalidate "all responses that might have discussed topic X" without searching the embedding space for entries near the topic's embedding.

This asymmetry creates a subtle problem. Suppose a company updates its product pricing policy. In an exact-match cache, you could search for entries whose keys contain the word "pricing" and remove them. In a semantic cache, you would need to embed a query like "product pricing policy", search for all cache entries with similarity above some threshold, and remove them. But this operation itself requires a similarity search that may be slow and may have recall errors: some relevant entries may not be found if their embeddings are below the search threshold.

One practical approach is to maintain a side index that maps semantic topics or document IDs to the cache entries that might depend on them. At cache store time, you classify the response by topic (automatically, using the LLM itself to tag the response) and add the entry ID to the topic's index. When a topic changes, you use the index to find and invalidate the relevant cache entries. Building and maintaining this index requires careful design but makes targeted invalidation tractable.

A simpler approach is to accept full flush as the invalidation strategy for semantic caches, while investing in fast cache warming to minimize the cold period. If your semantic cache can be warmed to 60% hit rate within 30 minutes of a flush (by replaying recent query logs), the cost of periodic full flushes may be acceptable.

A cache hit rate is the fraction of requests that were served from cache rather than by running inference. It is the primary metric for evaluating whether a caching strategy is working.

where:

• $\text{cache\_hits}$: the number of requests served from cache
• $\text{cache\_misses}$: the number of requests that required fresh inference
• $\text{hit\_rate}$: a value between 0 and 1 (or 0% to 100%)

A hit rate of 0 means the cache provided no benefit. A hit rate of 1 means every request was served from cache, and the model was never called after the initial warmup. In practice, well-designed caching systems often achieve hit rates of 20-60% for general-purpose chatbots and 70-90% for structured applications with narrow query spaces.

Hit rates are driven by the repetitiveness of your workload. Some systems have inherently cacheable traffic:

• Customer support bots where users repeatedly ask the same common questions
• Document summarization pipelines where the same documents are processed repeatedly
• Code generation assistants where the same boilerplate patterns appear frequently
• Content moderation pipelines where the same, or very similar, content is checked multiple times

Other systems have traffic that is inherently diverse:

• Open-ended creative writing assistants
• Systems with strong personalization where every response is user-specific
• Real-time systems that require fresh information on every request
• Conversational systems where multi-turn context makes each request unique

Understanding which category your system falls into determines how much effort to invest in caching. Attempting to apply semantic caching to an inherently diverse workload will produce a low hit rate at the cost of cache lookup overhead on every request, a net negative.

The depth of the query space is the key structural variable. A question-answering system with 50 well-defined topics and 1,000 possible questions per topic has a query space of roughly 50,000 questions. A system handling free-form creative requests has an effectively unbounded query space. The first system can achieve high cache hit rates with a moderate-sized cache; the second system cannot, regardless of cache size.

Aggregate hit rate is useful but can be misleading. A high hit rate driven by a small number of extremely common queries may coexist with very low hit rates for the long tail of queries that users care about most. Per-query-type or per-session hit rates give more actionable information.

Consider a customer support system where "reset password" accounts for 40% of all queries, and this query has a 100% cache hit rate. If the remaining 60% of queries cover hundreds of different issues and have 0% hit rate, the aggregate hit rate is 40%, which looks reasonable. But the users who came with complex issues received no caching benefit, and these users may matter disproportionately for satisfaction and cost.

For semantic caches, you should also measure the distribution of similarity scores at which cache hits occurred. A semantic cache with a high hit rate achieved mostly at similarities of 0.70-0.80 may be returning low-quality matches. A healthy distribution should show hits concentrated at high similarity (above 0.90) with the threshold positioned where the quality clearly degrades.

Another important metric is latency reduction. Even a modest hit rate can provide significant overall latency improvement if the cache hits occur on long, expensive inference calls. A 20% hit rate on a system where cached calls complete in 10 ms versus uncached calls in 3 seconds reduces average latency by nearly 70%.

To see this, let $h$ be the hit rate, $l_c$ be the cache lookup latency, and $l_i$ be the inference latency. The average latency is:

For $h = 0.20$, $l_c = 10\text{ ms}$, $l_i = 3000\text{ ms}$:

Without caching, $\bar{l} = 3000$ ms. The 20% hit rate delivers a 20% latency reduction in this scenario. Higher inference latency amplifies the benefit: a 10% hit rate on an 8-second inference call saves more absolute time than a 30% hit rate on a 500-ms inference call.

Hit rate is not the only cost metric. A cache miss followed by caching the result has a higher cost than a cache miss for a one-off query that will never be asked again. For rare queries, caching the response wastes storage and may cause evictions of more frequently used entries.

Selective caching addresses this by only caching responses that are likely to be requested again. You can estimate reuse probability from query frequency distributions, domain knowledge, or by requiring a minimum number of occurrences before caching. Frequency filtering, also called "minimum frequency caching", tracks how many times each query (or semantic cluster) has been requested and only adds it to the cache after the count exceeds a threshold. This avoids polluting the cache with one-off queries that consume capacity needed for high-frequency entries.

Building frequency estimation into your caching layer adds complexity but improves cache efficiency for large, diverse query spaces. The simplest implementation uses a count-min sketch (a probabilistic frequency estimation data structure) to track query frequencies with bounded memory. When the count for a new query crosses the threshold, the query is promoted to the cache.

The cold start problem is the flip side of selective caching. New queries need at least one uncached inference before they can be cached. During high-traffic events or product launches, many new queries arrive simultaneously, all missing the cache and all competing for inference capacity. This can create a feedback loop where high load leads to cache misses, which leads to higher inference load, which leads to even higher latency, reducing the chance that users rephrase and retry with a cacheable query. Warming before high-traffic events is the mitigation.

Natural language query distributions closely follow Zipf's law: if you rank queries by frequency, the $k$-th most popular query has frequency proportional to $1/k$. The most popular query is asked twice as often as the second most popular, three times as often as the third, and so on.

This power-law distribution has a direct implication for caching: a small cache covers a disproportionately large fraction of traffic. If the top 1% of distinct queries accounts for 50% of all requests (a typical ratio in Zipfian distributions), then a cache large enough to hold 1% of the distinct query space will serve 50% of all traffic. This is why semantic caches deliver value even when they store only hundreds of entries against a universe of millions of possible queries.

The Zipfian property also implies diminishing returns from cache size. The gain from increasing the cache from 100 to 200 entries is much larger than the gain from increasing it from 10,000 to 10,100 entries, because the low-rank entries that the larger cache adds are individually rare. The cache size that maximizes return on investment is usually much smaller than the size that maximizes hit rate.

Let's implement a semantic caching layer from scratch. We will build the core logic: embedding-based similarity search, cache storage, threshold-controlled retrieval, and hit rate tracking.

The pattern in the output reflects a property of real language workloads: a Zipfian distribution means a small cache covers a disproportionately large fraction of traffic. The first 50 entries in the cache often account for 40-50% of all requests. This is why semantic caches deliver value even when they store only hundreds of entries against a universe of millions of possible queries. The diminishing returns visible in the hit rate curve confirm that there is a natural stopping point for cache investment: beyond a certain size, additional entries cover only very rare queries that contribute little to overall performance.

The key parameters for the semantic caching system are:

• threshold: The minimum cosine similarity score required for a cache hit. Higher values (0.90-0.95) give precision; lower values (0.75-0.85) give recall. Domain-specific empirical tuning is necessary.
• ttl_seconds: How long a cached entry remains valid before expiration. Set based on how quickly the underlying information changes, not on system load.
• max_size: The maximum number of entries the cache stores before LRU eviction begins. Tuned against the Zipfian hit-rate curve for your workload.
• embedding_model: The model used to encode queries into vectors. Faster models reduce cache lookup latency but may produce lower-quality similarity estimates.

Caching is powerful but limited by the nature of LLM applications. The first and most persistent limitation is that not all tasks are cacheable. Creative generation, personalized content, and queries requiring fresh real-world information resist caching because no two requests have the same correct answer. Applying semantic caching to these tasks will produce responses that seem plausible but are contextually wrong, which can be more harmful than simply running inference. Before instrumenting a caching layer, you should audit your traffic to understand what fraction of requests are cacheable.

Semantic caching introduces a subtle failure mode: the embedding model and the generation model may not agree on what "similar" means. Two queries can be close in embedding space but require different responses. For example, "What is the boiling point of water at sea level?" and "What is the boiling point of water at high altitude?" are semantically similar (both about water's boiling point) but have different answers (100 degrees Celsius vs. roughly 90 degrees Celsius). An embedding model trained for general semantic similarity may assign these queries a high cosine similarity, causing the cache to return the wrong answer. Domain-specific embedding models and higher thresholds reduce but do not eliminate this risk. For safety-critical or high-stakes applications, you should validate the cache against a labeled dataset of known-similar and known-distinct query pairs before deploying it.

Prompt caching has a different set of limitations. It requires that the shared prefix be truly identical at the token level. Systems that inject dynamic content into system prompts, such as timestamps, request IDs, or per-user personalization strings, lose prompt caching benefits unless the dynamic content is carefully isolated at the end of the prompt. The discipline of ordering prompt components from most stable to least stable is not always compatible with other design goals, particularly in highly personalized systems where the personalization is conceptually part of the "setup" that comes before the user's question.

Cache invalidation remains an open engineering problem. The strategies described in this chapter, TTL, event-based invalidation, and version keys, all involve tradeoffs between freshness, complexity, and performance. Systems that serve frequently updated information must choose between serving stale cached responses and paying the full cost of inference on every request. There is no free lunch: the only way to guarantee freshness is to not cache, which means giving up the performance and cost benefits entirely. Good production systems typically accept a defined maximum staleness, communicate it to users where relevant, and design their cache TTLs to keep staleness within that bound.

A practical concern that often goes unaddressed in initial caching implementations is the effect of caching on model output diversity. Many LLM applications benefit from some variability in responses: slightly different wordings feel more natural to users and reduce the sensation of talking to a machine. Aggressive caching reduces this variability. If the same response is returned for every paraphrase of "what is the weather like?", conversations start to feel repetitive. For applications where conversational naturalness matters, you can introduce a randomization layer that occasionally bypasses the cache even for high-similarity matches, preserving some response diversity at the cost of a slightly lower effective hit rate.

Scalability is a final concern. An in-process Python cache of a few thousand entries (as implemented above) works for low-traffic prototypes. A production system with millions of cached entries distributed across many replicas requires a different architecture: a centralized vector database, a distributed key-value store for responses, and careful cache key partitioning to avoid hot spots. The embedding computation itself scales to dedicated inference endpoints if the volume requires it. As caching infrastructure becomes more complex, the operational overhead of maintaining it must be weighed against the cost savings it delivers.

Despite these limitations, caching consistently delivers some of the best return on investment of any latency optimization in LLM systems. In systems where even 10-20% of traffic can be served from cache, the cost and latency reductions are significant enough to change the economics of deployment. The engineering effort required to implement a basic exact-match or semantic cache is modest compared to the gains, making caching one of the first optimizations to add when moving from prototype to production.

Caching in LLM production systems operates at multiple levels, each targeting a different source of repeated computation.

Prompt caching works at the model level by reusing pre-computed KV tensors for shared prompt prefixes. It is transparent to application code and can reduce time-to-first-token by 80-90% for requests with long shared system prompts. The key requirement is placing stable content at the beginning of the prompt and keeping it exactly consistent across requests. Providers enforce minimum prefix lengths (typically 1,024 tokens) and use time-based eviction with windows of 5 minutes to 1 hour.

Semantic caching works at the application level by matching queries on semantic similarity rather than exact text equality. It uses text embeddings and vector similarity search to find cached responses for queries that are paraphrases of previously answered questions. The similarity threshold is the critical hyperparameter, balancing hit rate against the risk of returning incorrect cached answers. Context-aware caching handles the multi-turn case by including conversation history in the matching key.

Cache invalidation is the hardest part of caching. Time-based TTL is simple but blunt. Event-based invalidation is precise but requires dependency tracking. Version keys provide a clean mechanism for flushing stale entries when shared state changes. Semantic caches require special handling for invalidation because entries cannot be identified by deterministic keys, often requiring side indices or full-flush strategies.

Cache hit rates are the primary metric for evaluating caching effectiveness. Workloads following a Zipfian distribution, as most natural language workloads do, deliver disproportionate hit rates from small caches. Hit rate, combined with latency reduction and cost savings, provides the full picture of caching value. Aggregate hit rate alone can be misleading; per-query-type and similarity-score distributions give a sharper view of cache health.

The practical approach for production systems is to layer these mechanisms: exact-match caching for identical requests, semantic caching for paraphrases, and prompt caching at the model level for all requests. Together, these layers cover the most common sources of redundant computation and deliver meaningful improvements to cost, latency, and throughput. Monitoring, threshold calibration, and cache warming are the ongoing operational work that keeps each layer performing well as traffic patterns and content evolve.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about caching in LLM production systems.