Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

KV Cache Memory › Memory Visualization Across Models

Michael BrenndoerferJanuary 7, 20263 min read

Language AI HandbookAll chapters

Part 42 of 67Inference Optimization

Chapter 314KV Cache Memory › Memory Visualization Across Models

Reading section 9 of 14 · Memory Visualization Across Models · 3 minSection 9 of 14 · 3 min

Section 9 of 14 · 3 min read

KV Cache Memory › Memory Visualization Across ModelsOne section · Sections in this chapter9 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let us visualize how memory allocation differs across model sizes and architectures to build intuition for the relative importance of cache versus weights at different scales.

Out[24]:

Visualization

Stacked bar chart comparing memory allocation between model weights and KV cache across four model sizes. — Stacked bar chart comparing FP16 model weights and KV cache for four models at 8K context with batch size 1. Cache accounts for about 55% of combined weight-and-cache memory for GPT-2, 24% for LLaMA 7B, and 2% for the 70B models. Grouped Query Attention keeps the 70B cache small by using 8 KV heads instead of 64.

Without GQA, LLaMA 2 70B would require 8 times more KV cache memory, making long-context inference impractical on most hardware. The visualization clearly demonstrates how Grouped Query Attention keeps cache proportions manageable at the 70B scale, with the cache remaining a smaller fraction of total memory compared to what standard MHA would require. The architectural innovation of GQA is visible in the numbers: the 70B models with 8 KV heads have smaller caches than LLaMA 7B with 32 KV heads, despite being an order of magnitude larger in parameter count.