Sliding Window Attention › Key Parameters

When implementing or configuring sliding window attention, several parameters directly impact both performance and computational efficiency:

• window_size: The number of positions each token can attend to. Larger windows capture more context but increase computation. Common values range from 128-512 for efficient models to 4,096 for models like Mistral. Choose based on the expected range of relevant dependencies in your task.

• causal: Whether to use causal masking (True for autoregressive language models, False for bidirectional models like BERT). Causal masking ensures tokens cannot attend to future positions, which is required for text generation.

• dilation_rate: For dilated sliding windows, controls the gap between attended positions. A rate of 1 gives standard consecutive attention; rate 2 skips every other position, doubling the effective range. Use multiple dilation rates across heads for multi-scale attention.

• num_layers: The number of transformer layers affects the effective receptive field. With $L$ layers and window $w$, information can propagate approximately $L \cdot (w/2)$ positions in each direction. Deeper models can use smaller windows.

• d_k / d_head: The dimension of query and key vectors. Standard values follow $d_k = d_{\text{model}} / h$ where $h$ is the number of heads. This affects the scaling factor $\sqrt{d_k}$ in attention score computation.

• dtype_bytes: Memory precision for KV cache storage. Use 2 bytes for fp16/bf16 (recommended) or 4 bytes for fp32. Half precision halves memory usage with negligible quality impact.