KV Cache Compression › The Compression Opportunity

Before diving into specific techniques, we need to understand why compression is feasible in the first place. The basic question is this: can we discard information from the KV cache without significantly harming generation quality? To answer this, consider the attention patterns in a typical language model generating the 500th token. The query for this new token computes attention scores against all 499 previous key vectors. In a perfectly uniform distribution, each previous token would receive $1/499 \approx 0.2\%$ of the attention. In practice, attention is far from uniform, and this non-uniformity is precisely what makes compression possible.

The magnitude of this non-uniformity is striking. Empirical studies of transformer attention patterns consistently find that a small fraction of tokens, often 10-20%, account for 80-90% of the total attention weight. The remaining 80-90% of tokens share only 10-20% of the attention budget. This is reminiscent of the Pareto principle in economics: a small minority of inputs drives the majority of the output. For KV caches, this means that a compressed cache containing only the high-attention minority of tokens can approximate the full attention computation remarkably well, as long as we correctly identify which tokens belong to that necessary minority.

Studies of transformer attention patterns reveal several consistent phenomena that explain why some tokens matter far more than others:

• Local focus: Recent tokens often receive disproportionate attention, especially in lower layers. This makes intuitive sense because language exhibits strong local coherence, with adjacent words being grammatically and semantically related.
• Anchor tokens: Semantically important tokens such as subjects and verbs, along with tokens naming key entities, attract attention regardless of distance. These tokens carry the core meaning of a passage and remain relevant throughout generation.
• Attention sinks: As we covered in Part XVIII, initial tokens accumulate attention even when semantically irrelevant. This curious phenomenon appears to serve a computational purpose rather than a semantic one.
• Sparse activation: Many tokens receive near-zero attention and contribute minimally to the output. These tokens are prime candidates for removal without significant quality degradation.

This non-uniformity means that a cache storing only the high-attention tokens might preserve most of the information needed for accurate generation. The attention distribution is not merely slightly skewed; it is often dramatically concentrated on a small subset of tokens. The challenge lies in identifying these important tokens efficiently, without requiring expensive computation at each generation step. We need methods that can predict which tokens will be important for future generation steps, not just which tokens have been important in the past. As we will see, historical attention patterns turn out to be a surprisingly reliable predictor of future importance, at least for the tokens that truly matter.