KV Cache Memory › Limitations

The memory formula we derived is exact for a static, synchronous inference system where all sequences in a batch start and end together. Real production systems deviate from this idealized picture in ways that complicate the calculation.

The first major limitation is that our formula assumes every sequence in the batch occupies the same maximum sequence length. In practice, sequences have highly variable lengths. A batch of 8 requests might include one that generates only 100 tokens and another that generates 4,000. If you reserve the maximum possible cache for every sequence, you waste the memory allocated to short sequences. Paged Attention, covered in the next chapter, addresses this by dividing the cache into fixed-size pages and allocating only the pages each sequence needs. Our formula gives the worst-case upper bound; paged systems achieve much better average-case utilization.

The second limitation is that our activation memory estimate uses a rough multiplier that may not reflect the actual behavior of different model implementations or hardware backends. Activation memory depends on the specific operations being fused, the micro-batch size for pipeline parallelism, and the implementation details of the inference framework. Libraries like vLLM, TensorRT-LLM, and DeepSpeed all make different choices about when to materialize and deallocate activation tensors. For precise planning, measure actual memory consumption with your chosen model and inference framework on the target hardware rather than relying solely on our analytical formula.

A third limitation worth understanding is that our bottleneck analysis is simplified. The roofline model assumes all data movement is to and from the same level of the memory hierarchy, but GPUs have complex memory hierarchies including L1 cache, L2 cache, shared memory, and high-bandwidth memory. The effective bandwidth for small tensors may be much lower than the peak HBM bandwidth, and operations can be structured to exploit reuse in faster caches. FlashAttention, for example, achieves speedups precisely by tiling computations to fit in fast shared memory, reducing total HBM traffic even though the number of arithmetic operations remains the same. Our simplified analysis correctly identifies the direction of the bottleneck but underestimates the potential gains from careful memory access pattern optimization.

Finally, multi-GPU deployment introduces additional complexity that our single-GPU formula does not capture. When a model is sharded across multiple GPUs using tensor parallelism, each GPU holds a fraction of the model weights and computes a corresponding fraction of the attention heads. The KV cache must be partitioned accordingly, with communication between GPUs to synchronize attention outputs. This communication overhead adds to the effective memory bandwidth requirement and can shift the bottleneck analysis significantly. Sequence parallelism, where different sequence positions are processed on different GPUs, requires the KV cache to be gathered across all devices before each attention layer, creating collective communication costs that can dominate latency at large scale. These distributed system effects require separate analysis beyond the single-GPU formula we derived here.