Caching › Cache Hit Rates

A cache hit rate is the fraction of requests that were served from cache rather than by running inference. It is the primary metric for evaluating whether a caching strategy is working.

$$\text{hit\_rate} = \frac{\text{cache\_hits}}{\text{cache\_hits} + \text{cache\_misses}}$$

where:

• $\text{cache\_hits}$: the number of requests served from cache
• $\text{cache\_misses}$: the number of requests that required fresh inference
• $\text{hit\_rate}$: a value between 0 and 1 (or 0% to 100%)

A hit rate of 0 means the cache provided no benefit. A hit rate of 1 means every request was served from cache, and the model was never called after the initial warmup. In practice, well-designed caching systems often achieve hit rates of 20-60% for general-purpose chatbots and 70-90% for structured applications with narrow query spaces.

Hit rates are driven by the repetitiveness of your workload. Some systems have inherently cacheable traffic:

• Customer support bots where users repeatedly ask the same common questions
• Document summarization pipelines where the same documents are processed repeatedly
• Code generation assistants where the same boilerplate patterns appear frequently
• Content moderation pipelines where the same, or very similar, content is checked multiple times

Other systems have traffic that is inherently diverse:

• Open-ended creative writing assistants
• Systems with strong personalization where every response is user-specific
• Real-time systems that require fresh information on every request
• Conversational systems where multi-turn context makes each request unique

Understanding which category your system falls into determines how much effort to invest in caching. Attempting to apply semantic caching to an inherently diverse workload will produce a low hit rate at the cost of cache lookup overhead on every request, a net negative.

The depth of the query space is the key structural variable. A question-answering system with 50 well-defined topics and 1,000 possible questions per topic has a query space of roughly 50,000 questions. A system handling free-form creative requests has an effectively unbounded query space. The first system can achieve high cache hit rates with a moderate-sized cache; the second system cannot, regardless of cache size.

Aggregate hit rate is useful but can be misleading. A high hit rate driven by a small number of extremely common queries may coexist with very low hit rates for the long tail of queries that users care about most. Per-query-type or per-session hit rates give more actionable information.

Consider a customer support system where "reset password" accounts for 40% of all queries, and this query has a 100% cache hit rate. If the remaining 60% of queries cover hundreds of different issues and have 0% hit rate, the aggregate hit rate is 40%, which looks reasonable. But the users who came with complex issues received no caching benefit, and these users may matter disproportionately for satisfaction and cost.

For semantic caches, you should also measure the distribution of similarity scores at which cache hits occurred. A semantic cache with a high hit rate achieved mostly at similarities of 0.70-0.80 may be returning low-quality matches. A healthy distribution should show hits concentrated at high similarity (above 0.90) with the threshold positioned where the quality clearly degrades.

Another important metric is latency reduction. Even a modest hit rate can provide significant overall latency improvement if the cache hits occur on long, expensive inference calls. A 20% hit rate on a system where cached calls complete in 10 ms versus uncached calls in 3 seconds reduces average latency by nearly 70%.

To see this, let $h$ be the hit rate, $l_c$ be the cache lookup latency, and $l_i$ be the inference latency. The average latency is:

$$\bar{l} = h \cdot l_c + (1 - h) \cdot l_i$$

For $h = 0.20$, $l_c = 10\text{ ms}$, $l_i = 3000\text{ ms}$:

$$\bar{l} = 0.20 \times 10 + 0.80 \times 3000 = 2 + 2400 = 2402\text{ ms}$$

Without caching, $\bar{l} = 3000$ ms. The 20% hit rate delivers a 20% latency reduction in this scenario. Higher inference latency amplifies the benefit: a 10% hit rate on an 8-second inference call saves more absolute time than a 30% hit rate on a 500-ms inference call.

Hit rate is not the only cost metric. A cache miss followed by caching the result has a higher cost than a cache miss for a one-off query that will never be asked again. For rare queries, caching the response wastes storage and may cause evictions of more frequently used entries.

Selective caching addresses this by only caching responses that are likely to be requested again. You can estimate reuse probability from query frequency distributions, domain knowledge, or by requiring a minimum number of occurrences before caching. Frequency filtering, also called "minimum frequency caching", tracks how many times each query (or semantic cluster) has been requested and only adds it to the cache after the count exceeds a threshold. This avoids polluting the cache with one-off queries that consume capacity needed for high-frequency entries.

Building frequency estimation into your caching layer adds complexity but improves cache efficiency for large, diverse query spaces. The simplest implementation uses a count-min sketch (a probabilistic frequency estimation data structure) to track query frequencies with bounded memory. When the count for a new query crosses the threshold, the query is promoted to the cache.

The cold start problem is the flip side of selective caching. New queries need at least one uncached inference before they can be cached. During high-traffic events or product launches, many new queries arrive simultaneously, all missing the cache and all competing for inference capacity. This can create a feedback loop where high load leads to cache misses, which leads to higher inference load, which leads to even higher latency, reducing the chance that users rephrase and retry with a cacheable query. Warming before high-traffic events is the mitigation.

Natural language query distributions closely follow Zipf's law: if you rank queries by frequency, the $k$-th most popular query has frequency proportional to $1/k$. The most popular query is asked twice as often as the second most popular, three times as often as the third, and so on.

This power-law distribution has a direct implication for caching: a small cache covers a disproportionately large fraction of traffic. If the top 1% of distinct queries accounts for 50% of all requests (a typical ratio in Zipfian distributions), then a cache large enough to hold 1% of the distinct query space will serve 50% of all traffic. This is why semantic caches deliver value even when they store only hundreds of entries against a universe of millions of possible queries.

The Zipfian property also implies diminishing returns from cache size. The gain from increasing the cache from 100 to 200 entries is much larger than the gain from increasing it from 10,000 to 10,100 entries, because the low-rank entries that the larger cache adds are individually rare. The cache size that maximizes return on investment is usually much smaller than the size that maximizes hit rate.