KV Cache Compression › KV Cache Compression

As we explored in the previous chapters on KV cache fundamentals and memory analysis, the cache grows linearly with sequence length and can consume tens of gigabytes for long contexts. A model generating a 32,768-token response might accumulate several gigabytes of cached key and value tensors across all layers and attention heads. Paged attention addresses memory fragmentation and allocation overhead, but the total memory footprint of the cache itself remains substantial. KV cache compression tackles this problem directly by reducing the amount of data stored, either through selective eviction of less important cache entries or by quantizing the cached values to lower numerical precision.

The key insight driving compression research is that not all cached tokens contribute equally to generation quality. Attention patterns in transformers are highly non-uniform: certain tokens receive concentrated attention while others are nearly ignored across hundreds of generation steps. This sparsity creates an opportunity. If we can identify and preserve the high-value cache entries while discarding the rest, we can dramatically reduce memory usage with minimal impact on output quality. The central challenge is doing this identification accurately and efficiently, without expensive per-step computation that would offset the memory savings.

Think of the KV cache as a research library. When a scholar writes a paper, they do not continuously consult every book they have ever read. Instead, they repeatedly return to a small set of highly relevant sources, occasionally glance at recent acquisitions, and largely ignore the large middle ground of books that happened to be present on the shelves. A librarian who understood the scholar's research patterns could remove most of those rarely-consulted books without affecting the quality of the scholar's work. KV cache compression is the process of being that librarian: identifying which cached tokens are the books our model keeps returning to and discarding the rest.

This chapter covers the three main approaches to KV cache compression: eviction-based strategies that remove entire token representations from the cache, attention sink preservation that protects necessary initial tokens from eviction, and cache quantization that reduces the numerical precision of stored values. We examine the H2O (Heavy-Hitter Oracle) algorithm in detail as a representative eviction method, then explore how these techniques combine for maximum compression. We also analyze the quality implications of each approach and discuss the practical trade-offs involved in deploying compressed caches in production.

Understanding these techniques requires familiarity with attention mechanics from Part XIII and the KV cache fundamentals covered earlier in this inference optimization section. We will also build on the attention sink discussion from Part XVIII to explain why naive eviction strategies fail and how the research community solved this problem. By the end of this chapter, you will have a thorough understanding of how to reduce KV cache memory by more than 30x while preserving generation quality for the large majority of practical use cases.