Attention Sinks › Worked Example: Tracing One Eviction Step

A concrete walkthrough of a single cache eviction step makes the mechanics tangible. Suppose you are running a streaming inference session with 4 sink tokens and a window of size 8, giving a maximum cache size of 12. You have already processed 12 tokens, filling the cache. The cache currently holds positions 0, 1, 2, 3 (sinks) and positions 4 through 11 (the window). Now a 13th token arrives.

The cache is full, so something must be evicted. The eviction rule is: preserve all sink tokens, and drop the oldest window token. Position 4 is the oldest window token, so it gets evicted. After eviction, the cache holds positions 0, 1, 2, 3 (sinks) and positions 5 through 12 (the updated window). The cache is still 12 entries, and position 4 is gone permanently.

Here is what happens to the attention distribution during this step. Before eviction, the query for token 13 would compute attention scores against cache positions {0, 1, 2, 3, 4, 5, ..., 11}. The denominator in the softmax sums the exponentials of all 12 scores. After eviction, the query computes attention against {0, 1, 2, 3, 5, 6, ..., 12}: the sink tokens are present, and the new token (12) has been appended, but position 4 is absent. The denominator changes, but the sink tokens' contribution to it remains. The model behaves just as it was trained to: the sink positions absorb excess attention, the recent window provides local context, and the new token's generation is stable.

Now consider what would have happened with a pure sliding window (no sinks). The cache before eviction holds positions 2 through 13 (12 tokens). After eviction, it holds positions 3 through 13, and the positions 0 and 1 that once served as sinks are long gone. The attention denominator no longer includes any of the specialized sink positions. The probability mass that those positions used to absorb now pushes up attention weights on positions 3, 4, and 5, which were never trained to handle it. Hidden states computed from this distorted distribution propagate forward and corrupt the next computation. By token 100 of streaming, the error has compounded across many generations and the output is incoherent.

The worked example makes one more thing clear: the information content of the sink tokens is not what matters. Position 4, which we evicted, might have held a semantically rich word like "transformer" or "architecture." But we can discard it because the model can still make good predictions from positions 5 through 12 combined with the sink anchors at 0 through 3. By contrast, position 0 might hold something entirely unremarkable, like a BOS token or a formatting character, but we must keep it because its key vector has been shaped by training to stabilize the entire attention distribution.