Attention Sinks › Mathematical Analysis

To understand why attention sinks emerge and why they're essential for streaming inference, we need to examine the mathematics of attention itself. This walkthrough from the basic attention formula to the StreamingLLM solution will show a core tension in transformer design: the softmax function that makes attention work also creates the problem that sinks must solve.

The analysis proceeds in three stages. First, we examine the standard attention formula to identify exactly where the softmax constraint forces excess attention to accumulate somewhere. Second, we trace the training dynamics that direct that excess toward the first positions. Third, we follow the mathematics through a sink removal event to see precisely why and how the model breaks when sinks are absent. Each stage builds directly on the previous one, so by the end you will have a complete mechanistic account of the phenomenon from first principles.

One thing to keep in mind as you read through the formulas: the key tension is not unique to transformers. Any mechanism that must produce a normalized probability distribution over a set of candidates faces this pressure. The attention mechanism simply makes the problem particularly visible because the number of candidates (sequence positions) can be thousands deep, making the "spare" probability mass substantial even when the model is making perfectly good predictions about the relevant positions.

Self-attention is a weighted averaging mechanism. For each position in a sequence, the model asks: "Which previous positions should I draw information from, and how much from each?" The answer comes as a set of weights that sum to 1, forming a probability distribution over the context.

In a standard autoregressive transformer, the attention computation at position $t$ produces:

$$\text{Attention}(Q_t, K_{1:t}, V_{1:t}) = \text{softmax}\left(\frac{Q_t K_{1:t}^\top}{\sqrt{d_k}}\right) V_{1:t}$$

where:

• $Q_t$: the query vector at position $t$, representing what information this position is looking for
• $K_{1:t}$: the key matrix containing key vectors from positions 1 through $t$, where each key represents what information that position offers
• $V_{1:t}$: the value matrix containing value vectors from positions 1 through $t$, holding the actual content to be aggregated
• $d_k$: the dimension of the key vectors, used to scale the dot products and prevent them from growing too large

The softmax function is the critical piece. It takes the raw attention scores (dot products between queries and keys) and converts them into a probability distribution:

$$\sum_{i=1}^{t} \alpha_{t,i} = 1$$

where each individual weight is:

$$\alpha_{t,i} = \frac{\exp(Q_t K_i^\top / \sqrt{d_k})}{\sum_{j=1}^{t} \exp(Q_t K_j^\top / \sqrt{d_k})}$$

where:

• $\alpha_{t,i}$: the attention weight from query position $t$ to key position $i$, indicating how much position $t$ attends to position $i$
• $Q_t K_i^\top$: the dot product between query $t$ and key $i$, measuring their compatibility
• $\exp(\cdot)$: the exponential function, which ensures all values are positive
• The denominator sums over all $t$ positions, normalizing the weights to form a valid probability distribution

This is where the tension arises. The exponential function has a mathematical property that creates the sink phenomenon: $\exp(x) > 0$ for all finite $x$. No matter how irrelevant a position is, the model cannot assign it exactly zero attention. The softmax constraint forces the model to distribute some probability mass everywhere.

When the current token needs information from only a handful of positions, what happens to the attention weight that must go to the other 995 tokens in a 1000-token sequence? The model needs somewhere to put this "excess" attention, somewhere that won't disrupt the computation. During training, the first tokens become that destination.

The emergence of sink tokens is not designed but learned. During pretraining on billions of tokens, the model discovers that attending to early positions is "safe" for several reasons:

1. They're always present: Position 0 exists in every training example, so the model can reliably use it
2. Their representations stabilize early: By the time later positions attend to them, their hidden states have passed through many layers
3. Attending to them causes minimal harm: Mixing in a small amount of irrelevant but stable information is better than spreading that attention across many varying positions

Let $K_0$ denote the key vector at position 0 after training. This vector develops a distinctive property: for most query vectors $Q_t$, the dot product $Q_t K_0^\top$ is relatively high compared to random positions. The model has learned to make $K_0$ a universal "attractor" in key space.

To see why this specialization emerges, consider the gradient dynamics during training. Whenever the model generates a token that benefits from "dumping" excess attention, the gradient updates reinforce key-query alignment at position 0. Because this pattern occurs consistently across diverse training examples (position 0 is always present; the content at other positions varies wildly), the key vector $K_0$ experiences a stronger and more consistent gradient pressure than any other position. Over millions of training steps, this consistently pushes $K_0$ toward a direction in key space that maximizes dot products with a wide variety of queries.

Notice that this is a self-reinforcing dynamic. Once $K_0$ starts absorbing excess attention during training, the gradient updates make it even better at absorbing excess attention, which causes it to absorb even more, and so on. The model converges to a configuration where position 0 is a reliable sink precisely because the training process rewards that configuration. It is an emergent coordination mechanism: there is no explicit objective that says "make position 0 a sink," but the model discovers that arrangement independently as the globally stable solution to the softmax pressure problem.

The same mechanism applies, to a lesser extent, to positions 1 and 2. Each subsequent early position sees a consistent gradient pressure from all later positions, diminishing as we move further from the start. This explains the characteristic shape of the attention spike: highest at position 0, decaying quickly across the next few positions, then dropping to near-uniform levels for middle positions.

The heatmap shows that sink behavior is not an artifact of any single layer. Across all transformer layers, the first few positions consistently attract high attention weights. This cross-layer consistency suggests that sink tokens are part of how the model processes information, rather than a quirk of one particular layer's learned weights.

We can write the attention weight on position 0 using simplified notation:

$$\alpha_{t,0} = \frac{\exp(s_{t,0})}{\sum_{j=0}^{t-1} \exp(s_{t,j})}$$

where:

• $\alpha_{t,0}$: the attention weight from query position $t$ to position 0 (the first token)
• $s_{t,j} = Q_t K_j^\top / \sqrt{d_k}$: the scaled attention score between query $t$ and key $j$
• $\exp(s_{t,0})$: the numerator, which grows exponentially with how well query $t$ matches key 0
• $\sum_{j=0}^{t-1} \exp(s_{t,j})$: the normalization constant summing over all previous positions

For $\alpha_{t,0}$ to be consistently high across different queries and contexts, $s_{t,0}$ must be consistently large. The key vector $K_0$ evolves during training to project strongly onto the directions that queries typically occupy, regardless of the actual content at position 0.

Understanding why removal of sink tokens breaks the model requires following the math carefully. When you remove position 0 from the key-value cache, the attention computation changes in a specific and damaging way.

The new attention weights become:

$$\alpha'_{t,j} = \frac{\exp(s_{t,j})}{\sum_{k \neq 0} \exp(s_{t,k})}$$

where:

• $\alpha'_{t,j}$: the modified attention weight after removing position 0
• The denominator now sums only over positions $k \neq 0$, excluding the removed sink

The critical change is in the denominator. By removing $\exp(s_{t,0})$ from the sum, we've made the denominator smaller. Let's trace through what this means:

1. Numerators stay the same: For any remaining position $j$, its numerator $\exp(s_{t,j})$ hasn't changed
2. Denominator shrinks: The sum that divides the numerator is now missing the term $\exp(s_{t,0})$
3. All weights increase: Since numerator/smaller-denominator > numerator/larger-denominator, every remaining position receives more attention

The probability mass that previously went to position 0 must redistribute across the remaining positions. If the sink absorbed 30% of attention ($\alpha_{t,0} = 0.3$), that 30% now spreads across positions that weren't trained to receive it.

This redistribution causes two compounding problems:

1. Distribution mismatch: The new attention distribution doesn't match what the model saw during training. Each attention head has learned to produce useful representations given a specific expected distribution of weights. When that distribution changes, the representations become unreliable, leading to out-of-distribution hidden states that propagate through subsequent layers.

2. Cascade of new sinks: The model may try to use other early positions as sinks, but they lack the specialized key representation of position 0. As generations continue, the model keeps shifting which positions absorb excess attention, never stabilizing into a pattern it can use effectively.

StreamingLLM's insight is that you don't need the full history to maintain the attention distribution the model expects. You need exactly two things: the sink tokens to absorb excess attention, and recent context for actual language modeling. Everything in between can be discarded.

Formally, we maintain positions $\{0, 1, \ldots, k-1\}$ as sinks plus a sliding window $\{t-L, \ldots, t-1\}$ for recent context. The attention weight for any position $i$ in this combined set becomes:

$$\alpha_{t,i} = \frac{\exp(s_{t,i})}{\sum_{j \in \text{sinks}} \exp(s_{t,j}) + \sum_{j \in \text{window}} \exp(s_{t,j})}$$

where:

• $\alpha_{t,i}$: the attention weight from query position $t$ to position $i$
• $s_{t,i}$: the attention score between query $t$ and key $i$
• $\text{sinks} = \{0, 1, \ldots, k-1\}$: the set of $k$ preserved sink token positions
• $\text{window} = \{t-L, \ldots, t-1\}$: the sliding window of the most recent $L$ positions
• The denominator sums over both sets. This makes the attention weights form a valid probability distribution

The key insight is that the denominator structure matches what the model saw during training. The sink tokens contribute their $\exp(s_{t,j})$ terms to the normalization, absorbing their usual share of attention weight. The window tokens receive the context-relevant attention for actual language modeling. Because the sink tokens are present, the model operates in its training distribution rather than an out-of-distribution state.

This is why StreamingLLM works: it doesn't try to eliminate the sink phenomenon or work around it. Instead, it embraces the sink as a necessary component of how the model has learned to distribute attention, and simply ensures that component is always present.