Mistral Architecture › Sliding Window Attention

To understand sliding window attention, we need to first appreciate the problem it solves. Standard self-attention allows every token to attend to every other token. This seems natural: when predicting the next word, shouldn't the model consider the entire context? The answer is yes, in principle. But the computational reality tells a different story.

Consider what happens when you compute attention. For each token in your sequence, you calculate a score against every other token. With $n$ tokens, that's $n$ scores per token, and $n$ tokens total, giving $n \times n = n^2$ scores. Store these in memory, and you need space proportional to $n^2$. Compute them, and you need operations proportional to $n^2$.

This quadratic scaling creates a harsh reality:

Doubling the sequence length quadruples both memory and compute. By the time you reach 16K tokens, a common context length for modern models, you're dealing with hundreds of millions of attention scores per layer. Multiply by 32 layers and the numbers add up quickly.

Mistral's solution starts with a simple observation: in practice, tokens rarely need to attend to the entire preceding context with equal importance. Recent tokens matter more than distant ones for most language patterns. What if we formalized this intuition and restricted attention to only the most recent tokens?

This is exactly what sliding window attention does. Instead of allowing token $i$ to attend to all tokens from 0 to $i$, we restrict it to a fixed window of size $w$. Token $i$ can only attend to tokens in positions $\max(0, i - w + 1)$ through $i$, a span of at most $w$ tokens.

Think of it like looking through a sliding window that moves along with your position in the sequence. Early in the sequence, when you're at position 3, you might see positions 0, 1, 2, and 3. Later, at position 1000, you see positions 997, 998, 999, and 1000. The window always contains the same number of tokens (or fewer, at the start), regardless of how long the sequence grows.

Let's formalize this sliding window mathematically. We need a mask that tells the attention mechanism which positions are allowed (within the window) and which are forbidden (outside the window or in the future).

The attention mask $\text{Mask}_{i,j}$ determines whether query position $i$ can attend to key position $j$:

$$\text{Mask}_{i,j} = \begin{cases} 0 & \text{if } \max(0, i-w+1) \leq j \leq i \\ -\infty & \text{otherwise} \end{cases}$$

Let's unpack each component:

• $i$: the query position, meaning the token that's "asking" for information
• $j$: the key position, meaning the token that might provide information
• $w$: the window size (4096 for Mistral 7B)
• $\max(0, i-w+1)$: the leftmost position in the window, clamped to 0 for early tokens
• The condition $\max(0, i-w+1) \leq j \leq i$: defines the valid window, positions $j$ can attend to

The mask values themselves serve a specific purpose in the attention computation:

• $0$: When added to attention scores, leaves them unchanged, allowing attention
• $-\infty$: When added to attention scores, makes them negative infinity. After softmax, $e^{-\infty} = 0$, completely blocking attention

Why this particular formulation? The answer lies in how attention scores become attention weights. Raw scores pass through softmax, which exponentiates each score. By adding $-\infty$ to forbidden positions, we ensure their exponentiated values are exactly zero, not just small. This creates a hard boundary that the model cannot violate.

The power of sliding window attention becomes clear when we analyze its complexity. With full attention, each of $n$ tokens computes scores against all $n$ tokens, giving $O(n^2)$ operations. With sliding window attention, each of $n$ tokens computes scores against only $w$ tokens, giving $O(n \cdot w)$ operations.

When $w$ is fixed (4096 for Mistral), this becomes $O(n)$: linear in sequence length. The difference is significant:

• Full attention at 32K tokens: $32{,}768^2 \approx 1.07$ billion scores
• Sliding window at 32K tokens: $32{,}768 \times 4{,}096 \approx 134$ million scores

That's an 8x reduction at 32K tokens, and the gap widens as sequences grow longer.

The visual difference is clear. Standard causal attention fills the entire lower triangle, meaning late tokens must process attention scores for all preceding tokens. Sliding window attention creates a narrow band along the diagonal, greatly reducing the number of computations.

A natural concern arises: if each token can only see $w$ tokens back, how does the model capture long-range dependencies? If the sliding window is 4096 tokens, can a token at position 8000 ever learn anything from a token at position 100?

The answer is yes, and understanding why requires thinking about how information propagates through stacked transformer layers.

The Layered Propagation MechanismLink Copied

Consider a concrete example. At layer 1, token 8000 can directly attend to tokens 3905 through 8000 (a window of 4096 positions). It cannot see token 100 directly. But here's the key insight: token 3905 at layer 1 could see tokens 0 through 3905. When token 8000 attends to token 3905 at layer 2, it's attending to a representation that already contains information from the beginning of the sequence.

Each layer extends the reach:

1. Layer 1: Token $i$ sees positions $[i-w+1, i]$, a direct window of $w$ tokens
2. Layer 2: Token $i$ attends to tokens that themselves saw $w$ tokens back, extending indirect reach to $2w$
3. Layer $\ell$: Information can travel up to $\ell \times w$ positions through layered propagation

This gives us the receptive field formula:

$$R_\ell = \ell \times w$$

where:

• $R_\ell$: the receptive field at layer $\ell$, measuring how far back information can theoretically travel
• $\ell$: the layer number (1 to $L$)
• $w$: the sliding window size

Applying the Formula to MistralLink Copied

For Mistral 7B with $L = 32$ layers and $w = 4096$:

$$R_{32} = 32 \times 4096 = 131{,}072 \text{ tokens}$$

This theoretical receptive field of 131K tokens vastly exceeds Mistral's 8K context length. Even at layer 2, the receptive field is $2 \times 4096 = 8{,}192$ tokens, covering the entire context window.

This analysis reveals an important design choice: Mistral's window size and layer count are calibrated so that the receptive field exceeds the context length early in the network. By layer 2, any token can (indirectly) access information from any other token in the context.

The Trade-off: Direct vs Indirect AccessLink Copied

While the receptive field ensures information can flow across the entire context, there's a qualitative difference between direct and indirect access. When token 8000 directly attends to token 7999, it sees that token's representation with full fidelity. When it indirectly accesses token 100 through layered propagation, the information has passed through multiple aggregation steps, potentially becoming more diffuse.

For tasks requiring precise retrieval of specific early tokens, this indirect propagation may be less effective than full attention. But for most language modeling tasks, where context builds gradually and recent tokens matter most, the sliding window provides an excellent trade-off between efficiency and capability.

The visualization shows that by layer 2, the receptive field already exceeds the 8K context length. This means that even though individual attention operations are local, the overall architecture can still capture dependencies spanning the entire input.

Understanding the theory of sliding window attention is valuable, but the practical impact is what matters for deployment. Let's derive exactly how much memory and compute we save, starting from first principles.

Deriving the Memory FormulasLink Copied

The attention mechanism computes a score for each query-key pair. With $n$ query positions and $n$ key positions, we have $n \times n$ scores to store. Each attention head maintains its own score matrix, and each score requires storage space determined by the numerical precision.

For full attention, the memory required is:

$$M_{\text{full}} = n \times n \times h \times b = n^2 \cdot h \cdot b$$

For sliding window attention, each query position only attends to $\min(n, w)$ key positions. When the sequence is shorter than the window ($n \leq w$), we still compute $n$ scores per query. When the sequence exceeds the window ($n > w$), we compute exactly $w$ scores per query:

$$M_{\text{sliding}} = n \times \min(n, w) \times h \times b$$

Let's define each variable precisely:

• $M$: total memory in bytes for all attention score matrices
• $n$: sequence length (number of tokens)
• $w$: sliding window size (4096 for Mistral)
• $h$: number of attention heads (32 for Mistral)
• $b$: bytes per element (2 for fp16, 4 for fp32)

The Crossover PointLink Copied

The formulas reveal a key insight: the two methods behave identically when $n \leq w$. Both compute the same $n \times n$ attention matrix because the window hasn't started to "slide" yet. The divergence occurs when $n > w$:

• Full attention: Memory grows as $n^2$, accelerating as sequences lengthen
• Sliding window: Memory grows as $n \cdot w$, which is linear in $n$ since $w$ is fixed

For Mistral with $w = 4096$, the crossover happens at 4096 tokens. Below this, both methods are equivalent. Above this, sliding window provides increasing savings:

• At $n = 8192$: Sliding window uses $\frac{8192 \times 4096}{8192^2} = \frac{1}{2}$ the memory (50% savings)
• At $n = 32768$: Sliding window uses $\frac{32768 \times 4096}{32768^2} = \frac{1}{8}$ the memory (87.5% savings)

The savings percentage follows a simple formula: $1 - \frac{w}{n}$ when $n > w$. As sequences grow longer, the savings approach 100%.

Quantifying the SavingsLink Copied

Let's compute concrete memory numbers for Mistral's configuration: