Caching › Semantic Caching

Prompt caching operates within the model inference pipeline and is invisible to application code. Semantic caching operates at the application layer and requires explicit design decisions about what to cache, how to measure similarity, and when to return a cached answer versus generating a fresh one.

The core idea is simple: if a user asks "What is the capital of France?" and a later user asks "Which city is France's capital?", these questions have the same intent and the same answer. Why run a full LLM inference for the second request when you already have the answer? Exact-match caching would miss this case because the two strings differ. Semantic caching catches it by comparing meaning rather than text.

This capability matters because natural language is inherently paraphrastic. People express the same idea in hundreds of different ways. Customer support databases receive thousands of slight variations of the same common questions. Knowledge base assistants handle paraphrases of the same factual lookups. Without semantic caching, each variation requires a fresh model call even when the answer is already known.

The standard architecture for semantic caching uses text embeddings to measure query similarity. The intuition is that a well-trained embedding model places semantically similar texts close together in a high-dimensional vector space, regardless of surface-level phrasing differences.

When a new query arrives, you compute its embedding using an embedding model, search a vector store for the nearest cached query embedding, and return the cached response if the similarity exceeds a threshold.

The process works as follows. For each query that generates a fresh response, you store the query embedding and the response together in the cache. On subsequent requests, you compute the embedding for the new query and run a nearest-neighbor search over stored embeddings. If the closest match has a cosine similarity above your threshold, you return the cached response. Otherwise, you run inference and cache the new result.

This architecture separates the matching problem from the storage problem. The vector store handles similarity search efficiently, using approximate nearest-neighbor algorithms that can search millions of vectors in milliseconds. The cache store handles response retrieval, using a key-value store indexed by embedding identifier.

The choice of embedding model matters significantly. You want embeddings that capture semantic meaning precisely enough to distinguish questions with different answers, while being similar enough for paraphrases of the same question. Models like text-embedding-3-small, all-MiniLM-L6-v2, and specialized task-specific embeddings are common choices. The embedding model adds latency to each cache lookup, so small, fast models are preferred for latency-sensitive applications. A model that takes 100 ms to compute an embedding makes semantic caching impractical if your target uncached latency is 150 ms.

The similarity threshold is the most consequential hyperparameter in semantic caching. It controls the precision-recall tradeoff between cache hits and response quality.

A high threshold (say, 0.95) means only near-identical paraphrases match. You get very few false cache hits where semantically different questions receive the wrong cached answer. But the cache hit rate will be lower because slightly different phrasings do not match.

A low threshold (say, 0.70) means more queries match, increasing the hit rate. But you risk returning incorrect cached answers for queries that seemed similar but had different intent. A question about "the effects of caffeine on sleep" might incorrectly match "the effects of melatonin on sleep" at a low threshold, returning an answer about the wrong substance.

There is no universally correct threshold. The right value depends on the domain, the diversity of your query space, and the cost of a wrong answer. Factual question-answering over a narrow domain can tolerate lower thresholds because similar questions have similar answers. Creative or personalized tasks should use higher thresholds or disable semantic caching entirely.

The most rigorous way to set a threshold is empirical. Collect a labeled dataset of query pairs with human judgments of whether they have the same answer. Compute embedding similarities for all pairs. Find the similarity value that maximizes the F1 score between hit rate (recall) and precision (fraction of hits that are correct). This calibration should be redone whenever the embedding model or the knowledge domain changes significantly.

Basic semantic caching stores individual queries and their responses. A more structured approach is cluster-level caching, where you identify clusters of semantically similar queries and cache a canonical response for the cluster. New queries are matched to the nearest cluster rather than to individual cached queries.

Cluster-level caching has better space efficiency: instead of storing one response per unique query, you store one response per semantic cluster. It also provides more predictable behavior because the response for each cluster is explicitly defined rather than depending on which query happened to arrive first. If the first query in a cluster was phrased unusually and received an unusually phrased response, subsequent queries in the cluster will receive that unusual response. With cluster-level caching, the canonical response is deliberately crafted or quality-reviewed.

The tradeoff is that cluster-level caching requires upfront work to define clusters and their canonical responses. It works well for structured domains like customer support, where query types are well-defined and the set of clusters is relatively stable over time. It is harder to maintain for open-domain systems where new query types emerge continuously.

A hybrid approach that works in many production systems is to run cluster-level caching for the high-frequency query types that can be identified in advance, and fall back to individual query-level caching for the long tail. The top 50 customer support question types might be cluster-cached with curated answers, while the remaining diversity of queries uses standard semantic caching.

Semantic caching treats queries as independent. In a conversational system, this assumption breaks down. The meaning and appropriate response to "What do you mean?" depends entirely on the preceding conversation. "Tell me more about that" is meaningless without context. "Is it expensive?" requires knowing what "it" refers to.

A semantic cache that matches these queries to cached responses from different conversations will almost certainly return something irrelevant or misleading. The embedding for "Can you clarify?" is similar in every conversation, but the correct response is completely different in each case.

Context-aware caching addresses this by including conversation history in the cache key. The simplest approach is to embed the last few turns of conversation alongside the current query and match on the concatenated representation. This prevents false matches across different conversations at the cost of a lower cache hit rate, since the combined representation is more unique.

The depth of context to include involves a tradeoff. Including only the last turn is cheap and provides some context disambiguation. Including the last five turns provides much better disambiguation but makes the embedding more expensive to compute and the cached representation less reusable (since the probability of two sessions having identical recent history is very low).

An alternative approach that avoids this tradeoff is to restrict semantic caching to the first turn of a conversation, where there is no context to contaminate the match. The first user message in a conversation is most likely to be a direct, context-free question that matches well across sessions. Subsequent turns within a conversation are handled by exact-match caching keyed on the full conversation history, or by skipping caching entirely for multi-turn context.

At small scale, you can implement semantic caching with an in-memory list of embeddings searched by brute-force cosine similarity. This works for caches with a few thousand entries and single-server deployments.

At production scale, you need a vector database that supports efficient approximate nearest-neighbor search. Options include Pinecone, Weaviate, Qdrant, and Milvus for dedicated vector databases, as well as PostgreSQL with pgvector or Redis with RedisSearch for embedding support in existing infrastructure. The right choice depends on your existing stack, your query volume, and your tolerance for managing additional infrastructure.

The latency of the vector search must fit within your overall latency budget. A vector database query typically completes in 5-20 ms for collections of millions of vectors using optimized index structures like HNSW (Hierarchical Navigable Small World graphs). Adding the embedding computation time (5-30 ms depending on model size), the total cache lookup overhead is typically 10-50 ms. This is well worth the trade for requests that would otherwise take 1-10 seconds for inference.