KV Cache › Cache Memory Requirements

Understanding KV cache memory consumption is needed for deployment planning, as the cache often becomes the primary memory bottleneck in production systems. Unlike model weights that have a fixed size once the model is loaded, the cache grows dynamically during generation and can easily exceed the model weights in memory usage for long sequences or large batches. A system that can comfortably load and run a 7B model may be unable to serve multiple concurrent users at long context lengths, not because the model is too large, but because the caches are.

To derive the memory requirements, we must account for all the tensors that constitute the complete cache. For a single sequence of length $T$, we need to store keys and values for every layer and every head. The basic memory formula is:

$$\text{Memory} = 2 \times L \times H \times d_h \times T \times \text{bytes per element}$$

where:

• $2$: accounts for storing both keys and values, as each requires a separate tensor
• $L$: the number of transformer layers, each maintaining its own independent cache
• $H$: the number of attention heads per layer
• $d_h$: the dimension of each head, determining the size of individual key and value vectors
• $T$: the sequence length in tokens, the dynamic factor that grows during generation
• $\text{bytes per element}$: the memory size required for a single floating-point number (e.g., 2 bytes for FP16, 4 bytes for FP32)

Since the total model dimension is typically expressed as $d_{\text{model}} = H \times d_h$, we can simplify this formula by substituting:

$$\text{Memory} = 2 \times L \times d_{\text{model}} \times T \times \text{bytes per element}$$

where:

• $d_{\text{model}}$: the total model dimension, equal to $H \times d_h$
• $L$: the number of layers in the model
• $T$: the sequence length
• $\text{bytes per element}$: the numerical precision in bytes

This simplified formula reveals an important insight: the cache memory scales with the product of model depth and width, multiplied by sequence length. Doubling any of these factors doubles the memory requirement. This linear relationship is the cache's great advantage over the naive quadratic scheme, but it still produces very large absolute values for modern models.

To make these formulas concrete, let's calculate the cache requirements for a widely deployed model. The LLaMA 2 7B model has the following specifications:

• $L = 32$ layers
• $d_{\text{model}} = 4096$
• Typically stored in FP16 (2 bytes per element)

For a single sequence of length $T = 2048$, we can compute the cache size step by step:

$$\begin{aligned} \text{Memory} &= 2 \times 32 \times 4096 \times 2048 \times 2 \\ &= 1,073,741,824 \text{ bytes} \\ &= 1 \text{ GB} \end{aligned}$$

A single 2048-token sequence requires a full gigabyte just for the KV cache. This is separate from the memory needed for model weights and activations, along with the rest of the inference overhead.

For a batch of 8 sequences at the 4096-token context length, the memory requirement grows proportionally:

$$\begin{aligned} \text{Memory} &= 8 \times 2 \times 32 \times 4096 \times 4096 \times 2 \\ &= 17,179,869,184 \text{ bytes} \\ &= 16 \text{ GB} \end{aligned}$$

For comparison, the model weights themselves require about 14 GB in FP16. This means that for a batch of 8 long-context requests, the KV cache consumes more memory than the entire model. At longer context lengths or larger batch sizes, the cache can substantially exceed the model size, making memory management the necessary challenge for deployment.

Larger models and longer contexts exacerbate the memory pressure, creating significant challenges for deployment at scale. The following table shows KV cache sizes for various model configurations, illustrating how the requirements grow across different architectural choices:

The cache memory scales linearly with both model size (via the $L \times d_{\text{model}}$ product) and context length. This linear scaling in context length, combined with the quadratic scaling of the attention computation itself, makes long-context inference particularly challenging. A model like GPT-4 with a 128K context window requires hundreds of gigabytes per sequence, necessitating distributed systems and advanced memory management strategies. Understanding this table should give you a visceral sense of why every modern LLM serving system devotes significant engineering effort to cache optimization.