Attention Sinks › The Discovery: Why First Tokens Hoard Attention
Language AI HandbookAll chapters Attention Sinks › The Discovery: Why First Tokens Hoard AttentionOne section · Sections in this chapter2 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The attention sink phenomenon was first systematically analyzed in the StreamingLLM paper by Xiao et al. (2023). Researchers observed that in autoregressive language models, the very first tokens in a sequence consistently receive high attention weights from all later positions, even when those initial tokens carry no special semantic meaning.
An attention sink is a token position that accumulates disproportionately high attention weights across many query positions, typically regardless of the token's actual content or relevance. In autoregressive transformers, the first few tokens often serve as attention sinks due to softmax normalization requirements.
Consider what happens when a model processes a long sequence. For each new token, the self-attention mechanism computes attention weights over all previous positions. These weights must sum to 1 due to the softmax normalization. When the model encounters positions that aren't particularly relevant to the current prediction, it still needs to distribute some probability mass. The first tokens become a convenient "dump" for this excess attention.
This behavior emerges from training dynamics, not explicit design. During pretraining, models learn that attending to initial tokens is "safe" because they're always present and their representations stabilize early. The first token position essentially becomes a learned bias term that absorbs attention weight that would otherwise need to be spread across irrelevant positions.
Notice that the content of those first tokens is largely irrelevant to the sink function. The Xiao et al. study found that adding a dedicated newline character, a period, or even a blank token at position 0 produced an effective sink, because the model's key vectors at that position adapt during training to maximize their "catchall" property rather than to represent any particular linguistic meaning. The sink is a role, not a token type. The first position gets pressed into service because it is the position with the longest and most consistent training signal: every sequence in the pretraining corpus starts at position 0, so the key vector at position 0 accumulates gradient updates from more contexts than any other position.
The phenomenon extends across the model. Attention sinks are not confined to a single transformer layer or a single attention head. Visualizing attention maps across all layers shows the same pattern repeating: the first few positions receive high attention weights at virtually every layer and in the majority of heads. This cross-layer consistency indicates that attention sinks are woven into the model's overall information processing strategy, not a surface-level artifact of the final prediction head. The sink pattern stabilizes the hidden representations that flow between layers, because downstream layers receive slightly diluted but predictable signals from a stable upstream attractor.
The observation that transformers concentrate attention on early positions predates StreamingLLM. Researchers studying model interpretability noted "attention head specialization" as far back as 2019, with some heads appearing to copy information and others appearing to aggregate over long spans. What was missing was the practical insight: instead of viewing sinks as an interpretability curiosity, Xiao et al. asked whether they could be exploited for memory-efficient streaming. That reframing, from quirk to feature, produced one of the simplest and most practically impactful LLM inference optimizations published in 2023.
The visualization shows the characteristic shape: a sharp spike at the beginning of the sequence (the sink tokens), low uniform attention across the middle positions, and slightly higher attention weights on recent tokens that provide immediate context.
To understand just how much attention concentrates in the first few positions, let's look at the cumulative attention distribution. By summing the attention weights from left to right, we can see what fraction of total attention is captured by the first tokens.
The cumulative view highlights the sink phenomenon. The first 4 tokens alone capture about 29% of all attention weight, despite representing only 4% of the sequence positions. This concentration explains why removing these tokens causes such dramatic failures: you're removing the positions that the model relies on most heavily.