KV Cache › Limitations and Practical Considerations

KV caching introduces several challenges that drive ongoing research in inference optimization. These limitations directly motivate major inference optimization techniques that have emerged since 2020, from grouped-query attention to paged attention to speculative decoding.

The most basic limitation is that memory consumption scales linearly with sequence length. While this is better than the quadratic scaling we would face without caching, it still creates a hard constraint on context length and batch size. A server with 80 GB of GPU memory might fit the model weights comfortably at 14 GB, but may be unable to serve multiple long-context requests concurrently because the caches fill the remaining 66 GB faster than expected. This tension between throughput and context length is basic to LLM deployment and not solvable by any single optimization. The KV cache memory is the single most important factor determining how many requests a given GPU can serve simultaneously. Every percentage reduction in cache memory directly translates to higher concurrency and lower cost per query.

Memory fragmentation becomes increasingly problematic as the number of concurrent requests grows. When different sequences in a batch have different lengths, naive approaches either waste memory through padding or sacrifice batching efficiency. Consider a batch where one sequence is actively using 3,000 tokens of cache and another finished early and freed 2,000 tokens. A new incoming request wanting 2,500 tokens cannot use the freed space because it is not contiguous. The server either rejects the request, queues it until more contiguous space is available, or wastes memory by padding smaller requests to the maximum. Production systems must carefully manage cache allocation to maximize GPU utilization. The Paged Attention chapter (coming next) addresses this with a memory management approach inspired by operating system virtual memory, dividing the cache into fixed-size pages that can be assigned non-contiguously.

The cache must persist across the entire generation process. Unlike model weights that are read-only during inference, the KV cache is constantly growing and being read from. This means the cache cannot be easily offloaded to CPU during generation without incurring significant latency for memory transfers (CPU-GPU bandwidth is typically 10-30 GB/s, while GPU memory bandwidth is 1-3 TB/s, a gap of two orders of magnitude). Systems that need to handle many concurrent requests must carefully orchestrate which caches are active on GPU at any moment. Some systems implement "cache swapping" that moves inactive caches to CPU memory between generation steps, accepting the bandwidth cost in exchange for higher concurrency. This tradeoff is only viable when GPU processing time per step is long enough to hide the transfer latency.

A fourth, less obvious limitation is that the decode phase becomes memory-bandwidth-bound rather than compute-bound. During decode, each step processes a single token's worth of activations through the model's projection matrices. Modern GPUs have large FLOPs but moderate memory bandwidth. The weight matrices for a large model contain billions of parameters, and at each decode step the model must read in effect all of them from DRAM to compute a single output token. For LLaMA-2 7B, this means reading ~14 GB of weights and up to several GB of cache per generated token. The GPU's arithmetic units are starved waiting for memory, resulting in low hardware utilization. This is why batching is so important: serving 32 concurrent requests at once amortizes the weight-reading cost over 32 tokens, sharply improving utilization. The cache then becomes the bottleneck that limits how large a batch can be.

Grouped-Query Attention (GQA), which we discussed in Part XXIX, directly addresses the cache size problem by having multiple query heads share a single key-value head. For example, if 8 query heads share 1 KV head, the cache is 8 times smaller while the model's representational capacity for queries is unchanged. LLaMA-2 70B uses GQA with a 2:1 reduction (8 KV heads for 64 query heads), cutting the cache size by 8 times. This architectural choice was motivated specifically by the memory constraints we have described here: making the cache smaller is the most direct way to let larger batches or longer contexts.