KV Cache Memory › Summary

KV cache memory follows a simple but relentless formula: $2 \times L \times B \times T \times H_{\text{kv}} \times D_h \times \text{bytes}$. This linear scaling with batch size and sequence length means the cache can easily exceed model weights for long-context or batched inference. Understanding this formula and its implications is the foundation for every optimization decision in the chapters that follow.

Key takeaways:

• Model size scales differently than cache size: A 10x larger model does not require 10x more cache when using GQA. LLaMA 2 70B has a smaller cache than LLaMA 7B due to its 8 KV heads.
• Grouped Query Attention reduces cache size: GQA reduces the 70B cache by 8x compared to standard MHA, making large model deployment feasible on single GPUs.
• Long contexts are memory-expensive: At 32K or more tokens, KV cache typically exceeds model weight memory, fundamentally changing deployment requirements.
• Generation is memory-bound at typical context lengths: GPU compute utilization is limited by memory bandwidth when reading cached keys and values, not by arithmetic throughput.
• Deployment planning requires careful calculation: Before serving a model, estimate the memory used by the weights and cache, then add the remaining runtime overhead. Run the calculation for your worst-case sequence length and batch size.
• Data type is a controllable lever: Switching from FP16 to INT8 cache halves memory requirements with minimal quality impact, effectively doubling achievable batch size or context length on the same hardware.

Understanding these memory dynamics is essential for everything that follows in this section of the handbook. The next chapter explores Paged Attention, which applies virtual memory concepts to manage KV cache more efficiently and recover the utilization that static allocation wastes. Subsequent chapters cover cache compression techniques that reduce memory by quantizing or pruning cached values. Together, these techniques enable longer contexts and larger batches while keeping memory requirements manageable.