Attention Sinks › Why Removing Initial Tokens Breaks Generation

This puzzle shows the importance of attention sinks. Suppose you want to do streaming inference: process a very long document one chunk at a time, discarding old tokens to save memory. A natural approach is to keep a sliding window of the most recent $L$ tokens.

But this fails catastrophically. When you remove the first tokens, the model loses its attention sinks. The excess attention weight that previously flowed to position 0 must now go somewhere else. The model wasn't trained with this attention distribution, so it produces outputs that don't match any pattern it learned during pretraining.

The key insight is that the model's behavior at inference time is always conditioned on the distribution of attention it saw during training. Every attention head learned its weights, every layer learned its transformations, under the assumption that a certain fraction of attention would flow to the sink positions. When you remove those positions, you are not simply asking the model to "forget" those tokens. You are scrambling the normalization constant inside every softmax operation. What looked like 30% of attention going to semantically useful tokens now becomes the model's entire distribution, normalized over a strictly smaller set, producing inflated weights on positions that were never meant to receive them.

In practice, the failure mode follows a characteristic arc. For the first few tokens after the sinks are removed, the model may still produce plausible output, because the recent context window provides enough signal to overcome the distribution mismatch. But as more tokens are generated without sinks, the accumulated error in attention distribution corrupts the hidden states layer by layer. The model begins generating repetitive phrases, then grammatical non-sentences, then complete gibberish. This progression is not random. It reflects the model searching for new "safe" positions to serve as sinks, failing to find candidates with the right key-vector geometry, and progressively destabilizing its internal representations.

The failure is also not recoverable by longer windows. Even if you give the model a window of 10,000 recent tokens but remove the first few, quality degrades. The window size addresses a different problem (how much local context the model can use) than the sink problem (where the model routes excess attention weight). You need to solve both simultaneously to achieve stable streaming generation.

The contrast is clear. With sink tokens present (left), attention follows the pattern the model learned during training: high weight on the initial positions, with the rest distributed sensibly across recent context. Without sink tokens (right), the attention distribution becomes erratic. The model tries to find new sinks among the remaining tokens, but these positions weren't trained to serve that role.

In practice, this manifests as:

• Increased perplexity on downstream tokens
• Repetitive or looping generation
• Loss of coherence over long generations
• Complete degeneration into nonsense after enough tokens