KV Cache Compression › Compression Quality Analysis
Language AI HandbookAll chapters KV Cache Compression › Compression Quality AnalysisOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Compression introduces approximation error that can degrade generation quality, and understanding the nature of this error is needed for making informed trade-offs. Different compression techniques produce qualitatively different error patterns, which affects how they impact the attention mechanism and ultimately the generated text. Characterizing these error profiles helps you choose the right compression strategy for each deployment scenario.
The most important dimension along which the error profiles differ is their locality. Quantization introduces globally distributed errors: every retained token's representation is slightly perturbed, but none are completely lost. Eviction introduces locally catastrophic errors: the evicted tokens are completely gone, but retained tokens are perfectly represented. These different error distributions lead to different failure modes that are worth understanding in depth.
Let's visualize how different compression strategies affect the attention computation:
import numpy as np
# Simulate a realistic attention pattern
np.random.seed(42)
seq_len = 64
query_len = 1
# Create attention weights with structure:
# - High attention to initial tokens (sinks)
# - Local attention to recent tokens
# - Sparse attention to intermediate tokens
attention_logits = np.random.randn(query_len, seq_len) * 0.5
# Add attention sink
attention_logits[:, :4] += 2.0
# Add local attention
attention_logits[:, -8:] += 1.5
# Add some heavy hitter tokens
attention_logits[:, 15] += 1.8
attention_logits[:, 28] += 2.0
# Softmax to get attention weights
def softmax(x):
exp_x = np.exp(x - x.max(axis=-1, keepdims=True))
return exp_x / exp_x.sum(axis=-1, keepdims=True)
original_attention = softmax(attention_logits)[0]
# Simulate quantization error (small random noise)
quant_noise = np.random.randn(seq_len) * 0.01
quantized_attention = softmax(attention_logits + quant_noise.reshape(1, -1))[0]
# Simulate eviction: only keep sink (4) + heavy hitters (4) + recent (8)
keep_mask = np.zeros(seq_len, dtype=bool)
keep_mask[:4] = True # Sink
keep_mask[-8:] = True # Recent
keep_mask[15] = True # Heavy hitter
keep_mask[28] = True # Heavy hitter
evicted_attention = np.zeros(seq_len)
evicted_logits = attention_logits.copy()
evicted_logits[:, ~keep_mask] = -1e9 # Mask evicted tokens
evicted_attention_computed = softmax(evicted_logits)[0]
evicted_attention[keep_mask] = evicted_attention_computed[keep_mask]
The visualization reveals the distinct error profiles of each compression technique, illustrating a basic difference in how they affect the attention computation. Quantization introduces small, distributed errors across all positions, preserving the overall attention pattern while adding noise uniformly. The relative importance of tokens remains intact, and the model can still attend to the information it needs, albeit with slight imprecision. This graceful degradation makes quantization a relatively safe compression technique.
Eviction, by contrast, creates complete information loss for removed tokens. The error is not small and distributed but rather binary: retained tokens have zero error while evicted tokens have error equal to their original attention weight. The key to successful eviction is so that evicted tokens would have received near-zero attention anyway. When this condition holds, the total attention error is minimal because we are only losing tokens that contributed negligibly. When eviction removes a token that the model would have attended to significantly, generation quality suffers because that information is permanently lost.
The H2O algorithm's tracking of cumulative attention helps identify which tokens can be safely removed by giving evidence of their historical importance. Tokens that have consistently received minimal attention are unlikely to suddenly become necessary, making them safe eviction candidates. However, this heuristic is not perfect: a token that received little attention during the first 500 generation steps might become important at step 501 if the generated content shifts to reference that earlier context. This is the basic limitation of history-based eviction that no algorithm can fully overcome without knowledge of future generation steps.