Attention Sinks › Attention Sinks

When you deploy a language model for streaming generation, something strange happens. After processing a few thousand tokens, the model's output quality degrades. It starts repeating itself, loses coherence, and eventually produces gibberish. This failure mode isn't about running out of memory or hitting a hard context limit. It's about a subtle architectural quirk: the first few tokens in a sequence absorb a disproportionate amount of attention weight, regardless of their semantic relevance.

These overloaded positions are called attention sinks. Understanding why they exist and how to work around them makes streaming inference practical: models can generate coherent text indefinitely without quality degradation.

To appreciate why this quirk matters, consider how self-attention works. Every token in a transformer attends to every previous token, and the resulting attention weights must sum to exactly 1 due to the softmax normalization. When a model processes a 4,000-token document and generates the 4,001st token, it computes a probability distribution over all 4,000 prior positions. In theory, most of that weight should flow to semantically relevant positions. In practice, a substantial fraction always ends up on the very first token, even when that token is something as inert as a formatting character or a system prompt header. The model has no choice: softmax cannot assign zero weight to any position, so it routes "spare" probability mass to positions it has learned are safe to absorb it.

Think of attention sinks as pressure-release valves in the attention mechanism. When a hydraulic system has excess pressure, it routes the surplus through a designated low-resistance path rather than letting pressure build unpredictably. Attention sinks play the same role. They give the model a predictable place to deposit attention weight that has nowhere useful to go, keeping the remaining distribution clean and semantically meaningful. Remove the valve and the pressure distributes chaotically through paths that weren't designed to handle it.

This phenomenon was first systematically documented by Xiao et al. in their 2023 paper introducing StreamingLLM. Their key observation was deceptively simple: preserving just four initial tokens alongside a sliding window of recent tokens allowed language models to generate coherent text across sequences of arbitrary length. The memory footprint stays constant. The generation quality stays stable. The trick works because it preserves exactly the tokens the model's attention mechanism needs, even when those tokens carry no semantic content that would survive a reading comprehension test.

Understanding attention sinks requires working through three distinct questions. First, why does the attention mechanism develop these sink positions in the first place? Second, what goes wrong when you naively remove them during streaming? Third, how does StreamingLLM's cache design provide a principled solution that costs essentially nothing in memory? This chapter answers all three, building from the mathematics of softmax through a complete implementation of a streaming inference cache.