Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

KV Cache Memory › Sequence Length Effects

Michael BrenndoerferJanuary 7, 20265 min read

Language AI HandbookAll chapters

Part 42 of 67Inference Optimization

Chapter 314KV Cache Memory › Sequence Length Effects

Reading section 5 of 14 · Sequence Length Effects · 5 minSection 5 of 14 · 5 min

Section 5 of 14 · 5 min read

KV Cache Memory › Sequence Length EffectsOne section · Sections in this chapter5 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Sequence length has a linear effect on KV cache size, and this relationship emerges directly from our formula. The term $T$ (sequence length) appears as a simple multiplicative factor, meaning that doubling the context window doubles the cache memory. Unlike attention score computation, which grows quadratically with sequence length, cache memory grows strictly linearly. This linear relationship enables straightforward capacity planning and creates unavoidable memory costs for long contexts. The quadratic attention computation can be optimized away with techniques like FlashAttention; the linear cache growth cannot.

This linear relationship becomes particularly problematic for long-context models. The trend in language model development has been toward ever-longer context windows, from 512 tokens in early BERT models to 128K or even 1M token windows in modern systems. Every tenfold increase in context length requires tenfold more cache memory, creating a fundamental tension between capability and resource requirements. A system that works perfectly at 4K context can become infeasible at 32K context without any change to the model, hardware, or batch size. The context window is not a free architectural choice; every token of context costs memory.

Think of context length as the size of your desk. A small desk lets you see only the last few pages of a document. A large desk lets you spread out the entire manuscript. But a larger desk costs more, takes up more space in your office, and requires more effort to keep organized. A 128K context window is a desk the size of a tennis court. It provides extraordinary capability, but the cost in memory resources is proportionally extraordinary.

In[8]:

Code

# LLaMA 7B specifications
llama_7b = models["LLaMA 7B"]

# Calculate cache sizes for various context lengths
context_lengths = [512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072]
cache_sizes_gb = []

for ctx_len in context_lengths:
    cache = calculate_kv_cache_memory(
        num_layers=llama_7b["num_layers"],
        num_kv_heads=llama_7b["num_kv_heads"],
        head_dim=llama_7b["head_dim"],
        batch_size=1,
        seq_len=ctx_len,
        dtype_bytes=2,
    )
    cache_sizes_gb.append(cache["GB"])

# Model weight size for comparison
weight_gb = llama_7b["total_params"] * 2 / (1024**3)

Out[9]:

Visualization

Line plot showing KV cache memory increasing linearly with context length, crossing model weight threshold. — Linear scaling of KV cache memory for LLaMA 7B in FP16 with batch size 1 across context lengths from 512 to 131,072 tokens. Memory grows from 0.25 GB at 512 tokens to 64 GB at 128K tokens. The cache crosses the 13.0 GB model-weight reference near 27K tokens and becomes the dominant memory consumer beyond that threshold.

The crossover point, where KV cache exceeds model weights, is critical for deployment planning. This visualization clearly shows the linear relationship between sequence length and memory consumption, with the cache becoming the dominant memory consumer at longer contexts. Let us find that crossover point exactly:

In[10]:

Code

# Find crossover point for LLaMA 7B
crossover_seq_len = (
    weight_gb
    * (1024**3)
    / (
        2
        * llama_7b["num_layers"]
        * llama_7b["num_kv_heads"]
        * llama_7b["head_dim"]
        * 2  # bytes
    )
)

Out[11]:

Console

LLaMA 7B crossover point: 26,702 tokens
At this context length, KV cache equals model weights: 13.0 GB

This crossover point represents a major change in the memory profile of inference. Below this threshold, model weights dominate memory consumption and the cache is a relatively small overhead. Above this threshold, the cache becomes the primary memory consumer, growing linearly with every additional token while weights remain constant.

For context windows beyond approximately 27K tokens, the KV cache will consume more memory than the model itself, fundamentally changing deployment requirements and making cache optimization techniques essential rather than optional. This is why serving Claude at 100K context is categorically different, in infrastructure terms, from serving it at 4K context. The model has not changed. The hardware costs have multiplied.

For long-context applications such as document analysis, multi-turn conversations, and code completion over large codebases, the KV cache dominates memory usage. A 128K context window, as supported by models like Claude and GPT-4, requires 32 times more cache memory than a 4K window. This scaling explains why long-context inference often requires specialized infrastructure and optimization techniques that would be unnecessary for short-context applications.

The Crossover ThresholdLink Copied

The sequence length at which cache memory equals weight memory is a useful reference point for deployment decisions. Below the threshold, your hardware choice is driven primarily by model size. Above the threshold, you must think seriously about cache management strategies.

We can derive the crossover threshold analytically by setting the cache formula equal to the weight formula and solving for $T$ :

2 \times L \times B \times T \times H_{\text{kv}} \times D_h \times \text{bytes} = N_{\text{params}} \times \text{bytes}

where $N_{\text{params}}$ is the total parameter count. Solving for $T$ with $B = 1$ :

T_{\text{crossover}} = \frac{N_{\text{params}}}{2 \times L \times H_{\text{kv}} \times D_h}

where:

$T_{\text{crossover}}$ : the sequence length at which cache equals weight memory
$N_{\text{params}}$ : the total number of model parameters
$L$ : number of transformer layers
$H_{\text{kv}}$ : number of KV heads
$D_h$ : head dimension

For LLaMA 7B, this gives approximately 27,000 tokens. For LLaMA 2 70B with GQA, the reduced KV head count ( $H_{\text{kv}} = 8$ instead of 64 for full MHA) pushes the crossover threshold much higher, to roughly 427,000 tokens. GQA does not just save cache memory; it dramatically defers the context length at which cache starts to dominate, giving practitioners much more headroom before they need specialized cache management.