KV Cache Compression › Limitations and Practical Considerations

KV cache compression involves basic trade-offs that require careful consideration before deployment. Understanding these limitations is not merely academic: getting compression wrong can cause visible quality degradation that harms user experience or produces incorrect outputs in high-stakes applications. The limitations fall into several distinct categories, each requiring different mitigation strategies.

The most significant limitation is task sensitivity. Some tasks are highly sensitive to long-range dependencies that may be evicted or distorted by compression. Summarization requires synthesizing information from across the entire input document, meaning that evicting middle-context tokens can omit important facts from the summary. Question answering over long documents requires retaining the specific passage that answers the question, which may not have received high attention during the reading phase. Multi-hop reasoning requires preserving intermediate conclusions that serve as anchors for later steps. For all of these tasks, aggressive eviction can cause catastrophic failures where the model simply does not have access to the information it needs to produce a correct answer.

Eviction strategies face the challenge of predicting future importance from past attention. A token that received little attention during the first 500 generation steps might become important at step 501, but if it was evicted, that information is permanently lost. This temporal mismatch is particularly problematic for tasks where relevance depends on the specific query or generated content. In conversational settings, a user might refer back to something said many turns earlier, and the model's ability to respond accurately depends on whether that old context survived eviction. Heavy hitter detection helps, but cannot anticipate all future needs, particularly for out-of-distribution or unexpected queries.

Quantization interacts with numerical precision in subtle ways that can be difficult to predict in advance. While INT8 typically works well, INT4 can cause issues for models that rely on precise attention scores. Grouped query attention and multi-query attention, as we covered in Part XXIX, already reduce KV cache memory by sharing key and value heads across multiple query heads. Combining these architectural choices with aggressive quantization requires careful validation because the reduced dimensionality means each cached value carries more semantic load, which makes it more sensitive to quantization error.

Cache compression also complicates batched inference in non-obvious ways. Different sequences in a batch may have different important tokens, requiring either per-sequence eviction decisions (which is complex and adds overhead) or a union of important tokens across sequences (which is less aggressive compression and may exceed memory budgets). When one sequence in a batch has important tokens at positions 50-100 while another has them at positions 200-300, a shared eviction policy cannot serve both optimally. PagedAttention, covered in the previous chapter, handles memory allocation elegantly with non-contiguous blocks, but the combination of dynamic eviction with paged allocation adds implementation complexity that most production systems find challenging.

Finally, the computational overhead of compression must be measured rather than assumed negligible. Tracking cumulative attention scores requires gathering attention weights at each step, averaging across heads, and performing a running update. Identifying heavy hitters requires a partial sort or top-k operation on the score vector. Performing quantization requires computing per-tensor or per-channel statistics plus element-wise arithmetic. For latency-sensitive applications serving interactive users, the overhead from these operations may offset a significant fraction of the memory savings. Simple approaches like StreamingLLM's sink-plus-window strategy often provide the best latency-memory trade-off because they require no attention tracking and no score computation: the eviction policy is purely positional and runs in constant time. When throughput matters more than latency, the additional computation for sophisticated eviction is easier to justify because memory savings translate directly into larger batch sizes and higher tokens per second.

A practical guideline for deploying compression is to start with INT8 quantization alone, since it provides 2x memory reduction with minimal quality impact and nearly zero computational overhead on modern hardware with INT8 tensor core support. Then, if memory remains insufficient, add StreamingLLM-style eviction with a generous window to avoid long-range dependency failures. Only consider more aggressive H2O-style eviction after measuring quality on your specific task and confirming that the quality degradation is acceptable. This incremental approach avoids deploying compression that is more aggressive than necessary, keeping quality impact bounded while still achieving substantial memory reduction.