KV Cache › KV Cache
Language AI HandbookAll chapters KV Cache › KV CacheOne section · Sections in this chapter1 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When you ask a language model to complete the sentence "The capital of France is," it generates tokens one at a time: first "Paris," then perhaps a comma, then additional context. At each step, the model computes attention over all previous tokens to decide what comes next. Without optimization, this means the model recomputes the same attention calculations for "The," "capital," "of," "France," "is" every single time it generates a new token. For a 100-token response, that's nearly 5,000 redundant computations just for the prompt tokens. This cost does not just slow things down a little; it makes practical deployment nearly impossible.
The key-value cache, commonly called the KV cache, eliminates this redundancy by storing intermediate attention computations and reusing them across generation steps. This optimization reduces the computational complexity of autoregressive generation from quadratic to linear, making inference with large language models feasible at production scale. Think of the KV cache as the model's notepad: instead of re-reading the entire novel every time it needs to write the next sentence, the model keeps its reading notes and consults them directly. The notes grow with each sentence written, but the act of re-reading the full novel from scratch never happens.
Understanding the KV cache is necessary for reasoning about modern LLM serving. Paged attention, continuous batching, speculative decoding, and grouped-query attention all manage or extend the KV cache more effectively. Before you can reason about why these techniques matter, you need to understand what the cache is, why it works, what it costs, and where it breaks down.
This chapter begins by dissecting exactly where redundancy arises during autoregressive generation, building up from the attention formula itself. We then formalize how caching eliminates that redundancy, examine the per-layer and per-head structure of the cache in real architectures, and work through the memory arithmetic that determines whether a deployment is feasible. A fully worked numerical example shows each computation step-by-step before we move into a complete PyTorch implementation. We close with an honest examination of the limitations that have motivated years of follow-on research.
The KV cache is so basic that it is difficult to pinpoint a single paper that "introduced" it. Caching key-value pairs in autoregressive Transformers was a natural consequence of how the original Transformer architecture (Vaswani et al., 2017) was adapted for generation. The idea that intermediate representations could be reused across decoding steps appeared in early inference implementations of the original Transformer, and became standard practice as soon as researchers began deploying decoder-only models for text generation. GPT-2's publicly released code already included generation loops structured to exploit this property. The term "KV cache" became widespread as researchers began to study its memory costs formally, particularly with the rise of long-context models like GPT-3 and the subsequent explosion of 7B-to-70B open-weight models where cache memory began competing seriously with model weight memory for limited GPU resources.