Attention Sinks › Empirical Validation

Michael BrenndoerferUpdated July 4, 20253 min read
Language AI HandbookAll chapters
Part 18 of 67Long Context
Chapter 130Attention Sinks › Empirical Validation
Reading section 10 of 16 · Empirical Validation · 3 minSection 10 of 16 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 130 of 513

Section 10 of 16 · 3 min read

Attention Sinks › Empirical ValidationOne section · Sections in this chapter10 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The mathematical argument for StreamingLLM explains why the method works, but empirical confirmation is equally important. Two questions matter most in practice: how many sink tokens do you need, and how large does the window need to be? The answers determine the memory footprint of your deployment. The original paper's findings on these questions have been replicated across many model families.

The key metric for evaluating streaming generation quality is perplexity: the model's average uncertainty per token, measured on a held-out evaluation set. Lower perplexity means the model assigns higher probability to the correct next token, indicating better prediction quality. When we vary the number of sink tokens or the window size and measure how perplexity changes, we get a direct empirical measurement of how much each parameter affects generation quality.

Let's examine how perplexity changes as we vary the number of sink tokens and window size:

Out[21]:
Visualization
Line plot showing perplexity decreasing sharply from 0 to 4 sink tokens, then plateauing.
Perplexity versus number of preserved sink tokens. Model performance stabilizes after preserving 4 sink tokens, with minimal improvement from additional sinks.
Line plot showing perplexity decreasing as window size increases from 128 to 2048.
Perplexity versus window size with 4 sink tokens. Larger windows improve perplexity until the window captures sufficient local context, typically around 1024-2048 tokens.

The left plot confirms that preserving 4 sink tokens is sufficient for most models. Additional sinks provide diminishing returns. The right plot shows that window size matters too: larger windows give the model more recent context to work with, improving perplexity until it approaches the full-context baseline.