KV Cache Compression › Window-Based Eviction
Language AI HandbookAll chapters KV Cache Compression › Window-Based EvictionOne section · Sections in this chapter3 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The simplest eviction strategy maintains a fixed-size sliding window, keeping only the most recent tokens in the cache. This approach has clear appeal for several reasons. It is trivial to implement, requires no analysis of attention patterns, and guarantees constant memory usage regardless of sequence length. The underlying assumption is that recent context is the most relevant context, which holds true for many language modeling scenarios. The local attention patterns we observed earlier provide empirical support for this assumption: recent tokens reliably attract a significant portion of the attention budget.
Window-based eviction is analogous to a student who only reads the last paragraph they wrote before continuing. For tasks with strong local dependencies, this works well. For tasks requiring integration of information from across the entire document, it fails catastrophically. Understanding exactly where this boundary lies is important for knowing when to use window-based approaches and when to adopt more sophisticated strategies.
The implementation is straightforward: we simply maintain a buffer of the last key-value pairs and discard older entries as new tokens are generated. No attention tracking, no score computation, and no complex data structures are required. This simplicity makes it attractive for latency-sensitive deployment scenarios where the overhead of more sophisticated methods would be unacceptable.
def sliding_window_cache(keys, values, window_size):
"""
Maintains only the most recent window_size tokens in cache.
Args:
keys: (batch, heads, seq_len, head_dim)
values: (batch, heads, seq_len, head_dim)
window_size: Number of recent tokens to keep
Returns:
Truncated keys and values
"""
seq_len = keys.size(2)
if seq_len <= window_size:
return keys, values
# Keep only the last window_size tokens
return keys[:, :, -window_size:, :], values[:, :, -window_size:, :]import torch
# Demonstrate the sliding window
batch_size, num_heads, seq_len, head_dim = 1, 8, 100, 64
keys = torch.randn(batch_size, num_heads, seq_len, head_dim)
values = torch.randn(batch_size, num_heads, seq_len, head_dim)
window_size = 32
k_truncated, v_truncated = sliding_window_cache(keys, values, window_size)
memory_reduction_pct = (1 - window_size / seq_len) * 100Original cache: 100 tokens Window size: 32 Truncated cache: 32 tokens Memory reduction: 68.0%
Truncating to 32 tokens reduces cache memory by 68% in this example. The implementation is straightforward: we simply slice the tensor to keep only the final positions. However, this simplicity comes with a basic limitation: the approach discards all context beyond the window, including potentially important information from the beginning of the sequence.
Consider the practical implications through a concrete example. Imagine generating a response to a question. The question might appear at token positions 0-50, followed by supporting context at positions 51-400, with generation starting at position 401. A sliding window of 100 tokens would retain only tokens 301-400, losing the original question entirely. The model would have no memory of what it was asked, potentially creating irrelevant or incoherent responses. This illustrates why pure window-based approaches often fail for tasks requiring long-range dependencies. The approach works well for streaming applications or local language modeling, but breaks down when the task requires referencing distant context.