Attention Sinks › Key Parameters

When implementing StreamingLLM, four parameters govern the tradeoff between generation quality, memory usage, and task suitability. Getting these parameters right means understanding what each one controls and why it affects streaming behavior in the way it does. Fortunately, the empirical guidance from the original paper is clear enough that you can start with sensible defaults and tune conservatively from there.

The fundamental tradeoff is that larger parameter values (more sinks, larger window) produce better quality at the cost of more memory, while smaller values reduce memory at the cost of some quality degradation. For most practical deployments, the right balance lands in a relatively narrow range that the original paper characterized well.

The key parameters are:

• num_sink_tokens: Number of initial tokens to preserve as attention sinks. The default of 4 works well for most models, but you may need fewer (1-2) for smaller models or more (8-16) for models with unusual attention patterns. Start with 4 and adjust based on perplexity measurements.

• window_size: Number of recent tokens to keep in the sliding window. Larger windows capture more context but require more memory. Common values range from 512 to 2048. The right choice depends on the typical dependency length in your generation task: conversational AI may need 512-1024, while document summarization benefits from 2048+.

• max_cache_size: Total cache size, computed as num_sink_tokens + window_size. This determines the fixed memory footprint. For a 7B parameter model with 32 layers and 4096 head dimension, each token in the cache uses approximately 1 MB, so a cache of 2048 tokens requires about 2 GB.

• position_encoding_strategy: How to handle position encodings for window tokens. Options include keeping original positions (works with RoPE), resetting positions when sliding (may cause distribution mismatch), or using relative encodings (which are less sensitive to this mismatch). The choice depends on the base model's positional encoding scheme.

The position encoding strategy deserves special attention because it is the parameter most likely to cause subtle failures that are hard to diagnose. When a token at original position 5000 slides into the "first window slot" after many evictions, it still carries its position-5000 encoding if you are using absolute position embeddings. The model was trained to interpret position 5000 as a middle-of-document location, not as a near-past location. For models using RoPE (Rotary Position Embedding), keeping original positions works reasonably well because RoPE encodes relative distance rather than absolute position in a way that degrades gracefully. For models using absolute sinusoidal embeddings, you may need to recompute positions after each eviction to avoid systematic distribution mismatch.