KV Cache Memory › KV Cache Memory

In the previous chapter, we saw how the KV cache eliminates redundant computation during autoregressive generation by storing keys and values from previous tokens. This memory-compute tradeoff is fundamental to efficient inference: instead of recomputing key and value projections for every token at every generation step, we store them once and read them back later. The savings in computation are real and substantial. But every saved computation comes with a cost in memory, and that cost compounds quickly as sequences grow longer and models grow larger.

Understanding exactly how much memory the KV cache requires is essential for deployment planning. A seemingly simple question, "Can I run this model on my GPU?", requires knowing the model weight size and the memory the cache will consume during generation. The answer depends on the model architecture, the sequence length, the batch size, and the data type. Underestimate any of these factors and you face an out-of-memory error at runtime. Overestimate and you overpay for hardware. Getting this calculation right is not a minor implementation detail; it is the central skill that separates successful deployments from failed ones.

Think of the KV cache as a receipt that grows longer with every token you generate. At the beginning of a conversation, the receipt is short and fits easily in your pocket. As the exchange continues, the receipt extends, eventually spilling out onto the table and then onto the floor. Each line of the receipt must remain accessible because future computation depends on it. You cannot crumple up the early lines and throw them away without corrupting the attention mechanism. The GPU's memory is the pocket, and the question is always whether the receipt is going to overflow it before the conversation ends.

This chapter provides the mathematical tools to calculate these requirements precisely. We derive the exact formula for KV cache memory from first principles, examining how each architectural parameter contributes to the total. We then trace the effects of sequence length, batch size, and data type separately, building intuition for which factors matter most in different deployment scenarios. We close by analyzing the memory bottleneck that arises when cache growth dominates inference performance, and we connect that analysis to the optimization techniques covered in subsequent chapters.

The calculations in this chapter are surprisingly simple once you understand the structure of what is being cached. The formula involves only multiplication of a handful of architectural parameters. The insights, however, run deep: they explain why Grouped Query Attention exists, why long-context inference is so expensive, why serving many users simultaneously requires careful memory planning, and why GPU generation is almost always limited by memory bandwidth rather than arithmetic throughput.