KV Cache Compression › Key Parameters
Michael BrenndoerferJanuary 9, 20261 min read
Language AI HandbookAll chapters Part 42 of 67Inference Optimization
KV Cache Compression › Key ParametersOne section · Sections in this chapter11 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for KV cache compression are:
- sink_size: Number of initial tokens to always preserve in the cache. These attention sinks are necessary for stability and typically only 4 tokens are needed.
- window_size (or recent_size): Number of most recent tokens to keep. Ensures local context is available for coherent generation.
- heavy_hitter_size: Number of high-attention tokens to preserve based on historical importance. Larger values reduce eviction-induced quality loss at the cost of memory.
- bits: Target bit width for quantization (e.g., 4 or 8). Lower bits save more memory but may increase error. INT8 is almost always safe; INT4 requires validation.
The interaction between these parameters determines the memory-quality trade-off. A larger sink plus window budget allows more context but reduces compression. A higher bit width preserves more precision but uses more memory per token. Finding the right balance requires empirical evaluation on representative workloads rather than relying purely on theoretical analysis.