KV Cache Memory › Worked Example: Hand-Calculating LLaMA 7B Cache

Working through a concrete numerical example cements the formula and builds the intuition you need to estimate cache sizes quickly. Let us trace through the LLaMA 7B calculation step by step, so that every multiplication corresponds to a specific architectural decision.

LLaMA 7B has the following relevant specifications:

• $L = 32$ transformer layers
• $H_{\text{kv}} = 32$ key-value heads (standard MHA, no GQA in the original LLaMA 7B)
• $D_h = 128$ head dimension
• Inference in FP16: $\text{bytes} = 2$

Step 1: Memory per element. Each floating-point value in the cache occupies 2 bytes in FP16.

Step 2: Elements per head per token. Each head stores one key vector of length $D_h = 128$ and one value vector of length $D_h = 128$. That gives $128 + 128 = 256$ elements per head per token.

Step 3: Elements across all heads per token per layer. With $H_{\text{kv}} = 32$ heads, one layer stores $32 \times 256 = 8{,}192$ elements per token.

Step 4: Elements across all layers per token. With $L = 32$ layers, each token contributes $32 \times 8{,}192 = 262{,}144$ elements to the cache in total.

Step 5: Bytes per token across all layers. Multiplying by 2 bytes per element:

$$\begin{aligned} \text{Bytes per token} &= 2 \times L \times H_{\text{kv}} \times D_h \times \text{bytes} \\ &= 2 \times 32 \times 32 \times 128 \times 2 \\ &= 524{,}288 \text{ bytes} \\ &= 512 \text{ KB} \\ &\approx 0.5 \text{ MB} \end{aligned}$$

Step 6: Total cache for a sequence of 4,096 tokens. Multiplying by the sequence length:

$$\begin{aligned} \text{Total Cache} &= T \times \text{bytes per token} \\ &= 4{,}096 \times 524{,}288 \\ &= 2{,}147{,}483{,}648 \text{ bytes} \\ &= 2 \text{ GB} \end{aligned}$$

Step 7: Sanity check using the simplified MHA formula. Recall that for MHA, $H_{\text{kv}} \times D_h = D_{\text{model}}$. For LLaMA 7B, $D_{\text{model}} = 32 \times 128 = 4{,}096$. Plugging into the simplified formula:

$$\begin{aligned} \text{Cache} &= 2 \times L \times B \times T \times D_{\text{model}} \times \text{bytes} \\ &= 2 \times 32 \times 1 \times 4{,}096 \times 4{,}096 \times 2 \\ &= 2 \text{ GB} \end{aligned}$$

Both paths give the same answer, which is reassuring. At 4K tokens with a batch size of 1 in FP16, LLaMA 7B requires exactly 2 GB for its KV cache. This number is small enough to coexist with the model weights (roughly 14 GB in FP16) on a 24 GB GPU. But it doubles with every doubling of context length, and it multiplies linearly with batch size.

Step 8: Scaling to a different batch size. If we want to serve 8 concurrent requests (batch size 8):

$$\text{Cache for batch 8} = 8 \times 2 \text{ GB} = 16 \text{ GB}$$

Adding 14 GB for model weights and roughly 2 GB for other overhead, this already approaches the 32 GB limit of an A100 40 GB GPU, leaving only a small safety margin. This calculation, which takes under a minute once you know the formula, determines whether a deployment plan is feasible before you ever touch the hardware.