Attention Sinks › The StreamingLLM Solution

StreamingLLM proposes a simple fix: instead of a pure sliding window, always keep the first few tokens. The elegance of this solution is that it requires no retraining, no changes to model weights, and no modifications to the attention computation itself. It is purely a cache management strategy. You change which tokens you store and present to the attention mechanism, and the model's existing behavior takes care of the rest.

The observation that leads to the fix is worth stating explicitly. The problem with naive sliding windows is not that old tokens are discarded. It is that the specific tokens at positions 0 through 3 are discarded. Tokens at positions 50, 100, or 500 can be discarded without meaningful harm, because the model's attention mechanism was not specifically trained to rely on their presence at a particular position. The sink tokens at the very beginning, however, have developed key-vector representations that act as attractors for excess attention weight across the entire model. They play a structural role in keeping attention distributions normalized and predictable.

The attention mechanism context becomes:

By preserving just 1 to 4 initial tokens (the attention sinks), plus a sliding window of recent tokens, you maintain the attention distribution the model expects while bounding memory usage. The generation can continue indefinitely without quality degradation.

In practice, this means the key-value cache that a transformer maintains during generation has two regions instead of one. The first region is a small fixed buffer holding the key and value vectors of the sink tokens. This buffer never changes once the initial tokens are processed. The second region is a circular buffer holding the most recent $L$ key-value pairs, which shifts by one position each time a new token is generated. The total cache size is bounded by $k + L$ regardless of how many tokens have been generated in total, where $k$ is the number of sink tokens.

Think of it like a news anchor keeping permanent notes about the show's topic and guest background (the sink tokens) alongside a rolling transcript of the last few minutes of conversation (the sliding window). The notes are never thrown away because they anchor the context. The transcript slides forward as new dialogue arrives. Together they give the anchor enough to respond coherently to the current moment without needing to store every word ever spoken on the show.

The key insight is that you don't need the entire history. You need:

1. The attention sinks to absorb excess attention weight
2. Recent context for actual language modeling

Everything in between can be safely discarded without affecting generation quality.