Caching › Prompt Caching

Prompt caching exploits the internal mechanics of transformer inference. To understand why it saves computation, you need to understand what the model does when it processes a prompt.

When a transformer processes a sequence of tokens, each attention layer computes key and value matrices for every token in the input. These matrices encode each token's representation in the context of every other token it can attend to. Computing them requires matrix multiplications that scale with both the sequence length and the model's hidden dimension, making them the dominant cost in long-context inference.

The key and value matrices for a given token depend only on that token's position and all preceding tokens. They do not depend on what comes after. This property is what makes prefix reuse possible.

The essential observation is that if two prompts share a prefix, the key and value matrices for that prefix are identical in both requests. There is no mathematical reason to recompute them. Prompt caching implements exactly this optimization: the computed KV tensors for a prefix are stored in fast memory and reused whenever a new request shares that prefix, bypassing the most expensive portion of the forward pass.

The cost savings are substantial. A request whose prompt is 90% shared prefix and 10% new content requires the model to compute full attention only over the new tokens, dramatically reducing time-to-first-token (TTFT). For requests where the system prompt is 2,000 tokens and the user message is 50 tokens, prompt caching can reduce TTFT by 80-90%. The financial savings are equally large because most providers charge for input tokens processed, and cached tokens are processed at a significant discount or not charged at all.

Not all prompts can benefit equally from prefix caching. The benefit depends on how prompts are structured relative to the caching mechanism.

Prefix caching matches on the exact sequence of token IDs from the beginning of the prompt. If a shared component appears anywhere other than the beginning, it does not help. This is why prompt engineering for production systems often includes a discipline: place stable, reusable content at the front. A typical ordering runs from most stable to least stable:

• System prompt (instructions, persona, output format)
• Retrieved documents or context (may vary per query but often repeated across sessions)
• Few-shot examples
• The user's actual question

This ordering ensures that as many leading tokens as possible are shared across requests, maximizing the prefix that can be cached. Violating this ordering, for example by placing user-specific metadata at the beginning, can eliminate all caching benefit even when 90% of the prompt content is shared.

The granularity of caching is typically at the cache block level, where each block covers a fixed number of tokens (commonly 16 or 32). A prefix must be long enough to fill at least one complete block to be eligible for caching. Short system prompts of only a few tokens may not save as much as expected because the partial final block must still be recomputed. In practice, this means that very short system prompts gain little from prefix caching even when they are consistently shared.

Most providers require a minimum prompt length before activating prompt caching because the overhead of cache lookup and storage for very short prompts exceeds the savings. The cache lookup itself requires hashing the prefix, searching the cache index, and loading stored tensors from memory, all of which take time. For very short prefixes, this overhead may exceed the savings from not recomputing the prefix.

Anthropic's Claude API activates prompt caching at 1,024 tokens for Claude 3 models. OpenAI's context caching for GPT-4o activates at 1,024 tokens as well. Google's Gemini API uses a minimum of 32,768 tokens for its explicit caching feature, targeting use cases with very long shared context such as entire code repositories or long documents. These thresholds reflect each provider's judgment about where the overhead-to-savings ratio becomes favorable.

Understanding minimum lengths matters for system design. A system prompt that is 800 tokens long sits below the caching threshold and will be fully recomputed on every request regardless of how consistently it is shared. Padding to reach the threshold by adding more thorough instructions is sometimes worth doing.

Cached KV states are not stored indefinitely. They consume GPU memory, the scarcest resource in LLM deployments. A single cached context for a 2,000-token prompt in a large model can occupy tens of megabytes of GPU memory. Multiply this across thousands of distinct prefixes and the memory pressure becomes significant.

Providers implement time-based eviction: a cached entry is retained for a sliding window (typically 5 minutes to 1 hour) and evicted if no new request uses it within that window. The eviction policy resembles a least-recently-used (LRU) scheme at the time dimension: entries that have not been accessed recently are removed to free memory for new entries. Systems with bursty but infrequent traffic may see poor cache hit rates because the cached state is evicted between bursts.

This eviction behavior has practical implications for how you think about caching economics. A batch job that runs hourly and shares a long system prompt will likely miss the cache on every run unless you either increase the cache lifetime or keep the cache warm with periodic requests. Designing around eviction windows requires knowing your traffic pattern.

Some providers expose cache lifetime controls that allow you to trade memory usage for higher hit rates. A longer cache lifetime means higher hit rates for infrequent-but-repeated workloads at the cost of holding more GPU memory across longer periods. The right setting depends on your traffic burstiness and the ratio of cache storage cost to inference cost.

The standard prompt caching described above requires exact prefix matching at the token level. Two prompts share a cached prefix only if their token sequences are literally identical from token 0 through some boundary. A single changed character, even in whitespace, invalidates the match.

This strictness is intentional and mathematically necessary. The KV cache stores tensors that encode exact positional relationships. The key and value for a token at position $i$ in the attention computation depends on the token's embedding, the positional encoding for position $i$, and the learned weight matrices. Reusing a cached KV state for a slightly different prefix would produce incorrect outputs because the attention patterns would be based on wrong positional information.

The practical implication is that prompt templates must be constructed with care. Dynamic content such as timestamps, user IDs, session identifiers, or request-specific metadata should go at the end of the prompt, after all shared content. Even a date injected into the system prompt ("Today's date is March 15, 2025") will prevent caching if it changes between requests, because the token "March" at position 10 will have a different KV representation than "April" at the same position.

Systems that inject per-user personalization into the system prompt must decide whether to sacrifice caching for personalization or restructure their prompts to isolate the personalizing content at the tail. Often, the best solution is to move lightweight personalization information (name, preferences) into the final human turn rather than the system prompt, preserving the shared system prompt prefix.

Prompt caching benefits are visible in two metrics: TTFT and cost. TTFT is measured as the time from request submission to the first token of the model's response. Cost is measured in dollars per request.

For a prompt with $P$ prefix tokens and $N$ new tokens, where the prefix is fully cached, the effective input token count for billing is $N$ rather than $P + N$. If the cached token discount is $d$ (e.g., 0.10 for a 90% discount on cached tokens), the cost is:

$$\text{cost} = N \cdot c_{\text{input}} + P \cdot c_{\text{input}} \cdot d$$

where $c_{\text{input}}$ is the per-token input price and $d$ is the cached token fraction (near 0 for fully cached prefixes).

For TTFT, the savings come from skipping the forward pass over the prefix. The reduction is roughly proportional to the fraction of the prompt that is cached, because the forward pass time scales linearly with the number of tokens processed.