KV Cache Compression › Summary

KV cache compression addresses the memory bottleneck of long-sequence generation through two complementary approaches: reducing the number of cached tokens through eviction, and reducing the precision of stored values through quantization.

Window-based eviction provides the simplest implementation but sacrifices all long-range context. Attention sink preservation recognizes that initial tokens serve as computational anchors and must be protected from eviction to maintain generation coherence. The H2O algorithm extends this by tracking cumulative attention to identify and preserve "heavy hitter" tokens that consistently receive high attention, balancing recency with historical importance. The key insight across all eviction strategies is that attention patterns are highly non-uniform and temporally persistent, which makes historical attention a reliable proxy for future importance.

Cache quantization offers orthogonal memory savings without removing any tokens from the cache. INT8 quantization typically achieves 2x compression with negligible quality impact. INT4 provides 4x compression but requires per-channel quantization and careful validation, particularly for precision-sensitive tasks. The basic mechanism is the same as general quantization: represent continuous values with discrete integers, accepting bounded approximation error in exchange for dramatic storage reduction.

Combining eviction with quantization can achieve compression ratios exceeding 30x for long sequences, letting use cases that would otherwise require prohibitive memory. The two techniques combine multiplicatively because they address orthogonal dimensions: eviction reduces token count while quantization reduces bytes per token. The choice of compression strategy depends on the specific use case. Tasks requiring precise long-range reasoning may tolerate only light compression, while streaming applications or tasks with primarily local dependencies can use aggressive compression. Understanding these trade-offs enables you to select appropriate configurations that balance memory efficiency with generation quality.