Attention Sinks › Designing Effective Sink Tokens
Language AI HandbookAll chapters Attention Sinks › Designing Effective Sink TokensOne section · Sections in this chapter8 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Not all initial tokens make equally good sinks. The effectiveness of an attention sink depends on what token occupies that position during training. A token becomes a good sink when its key vector, after training, projects strongly onto a wide variety of query vectors. That property is acquired through repeated exposure: the more consistently a token appears at position 0 across diverse training examples, the more thoroughly the training process shapes its key representation into a universal attractor.
This has practical consequences for how you choose which tokens to preserve during streaming inference. The StreamingLLM paper tested several choices and found that the model's own BOS token is almost always the most effective single sink. Supplementary sinks can come from system prompt tokens if the model was instruction-tuned on conversations, or from learned sink tokens if the model was trained with explicit sink support. In all cases, the quality criterion is the same: how well does this token's key vector absorb excess attention across diverse query contexts?
Several strategies produce effective sinks, each with different implementation requirements:
Beginning-of-Sequence Token
Most tokenizers include a dedicated <BOS> (beginning-of-sequence) or <s> token that always appears at position 0. Because this token appears at the start of every training sequence, it develops a strong sink representation. The model learns that this token is always present and always available to absorb excess attention.
System Prompt Tokens
For instruction-tuned models, the system prompt often provides natural sink tokens. Phrases like "You are a helpful assistant" appear at the start of every conversation, making them reliable candidates for sink positions. The model has seen these tokens in position 0-20 millions of times, so their key representations have thoroughly adapted to the sink role.
Dedicated Sink Tokens
Some models are trained with explicit sink tokens. These are special tokens inserted at the beginning of sequences specifically to serve as attention sinks. During training, the model learns to route excess attention to these tokens rather than developing the behavior organically.
def count_sink_tokens_needed(model_perplexity_data, threshold=1.05):
"""
Determine the optimal number of sink tokens based on perplexity analysis.
When removing initial tokens, perplexity increases. We find the minimum
number of tokens to preserve such that perplexity stays within threshold.
Args:
model_perplexity_data: Dict mapping num_preserved_tokens -> perplexity
threshold: Maximum acceptable perplexity ratio vs. baseline
Returns:
Recommended number of sink tokens to preserve
"""
if not model_perplexity_data:
return 4 # Default recommendation from StreamingLLM paper
baseline_perplexity = max(model_perplexity_data.values())
for num_tokens in sorted(model_perplexity_data.keys()):
ppl = model_perplexity_data[num_tokens]
if ppl <= baseline_perplexity * threshold:
return num_tokens
return max(model_perplexity_data.keys())# Simulated perplexity data showing effect of sink token count
simulated_data = {
0: 45.2, # No sinks: very high perplexity
1: 18.7, # 1 sink: still elevated
2: 12.3, # 2 sinks: getting better
3: 10.1, # 3 sinks: close to baseline
4: 9.8, # 4 sinks: matches baseline
8: 9.7, # 8 sinks: marginal improvement
}
recommended = count_sink_tokens_needed(simulated_data)Perplexity vs Number of Preserved Sink Tokens ============================================= 0 sink tokens: perplexity = 45.2 <-- recommended 1 sink tokens: perplexity = 18.7 2 sink tokens: perplexity = 12.3 3 sink tokens: perplexity = 10.1 4 sink tokens: perplexity = 9.8 8 sink tokens: perplexity = 9.7
The data shows that perplexity drops sharply when preserving the first few tokens, then plateaus. With zero sink tokens, perplexity explodes to 45.2, indicating severe model degradation. Adding just one sink token cuts perplexity by more than half. By four sink tokens, perplexity reaches 9.8, nearly matching baseline performance. Additional sinks beyond four provide diminishing returns, matching the StreamingLLM paper's recommendation.