Qwen Architecture › Core Architectural Components

This section builds your understanding of Qwen's key innovations step by step. We'll start with the attention mechanism, where Qwen makes its most impactful efficiency choice, then explore the feed-forward network and positional encoding. For each component, we'll first build intuition about the problem being solved, then examine the mathematical formulation, and finally see it in code.

To understand why Qwen uses Grouped-Query Attention (GQA), we first need to understand the memory bottleneck in transformer inference.

The main bottleneck is KV cache growth.

During autoregressive generation, transformers must store the key (K) and value (V) vectors for all previously generated tokens. This "KV cache" allows the model to attend to its full history without recomputing everything at each step. For a model with $L$ layers, $h$ attention heads, head dimension $d_h$, and sequence length $n$, the cache requires:

$$\text{KV Cache Memory} = 2 \times L \times h \times n \times d_h \times \text{bytes\_per\_element}$$

For Qwen-72B generating a 4096-token sequence in FP16, this exceeds 40GB just for the cache. The insight behind GQA is that not every query head needs its own unique key-value representation, so we can share KV heads across groups of query heads to sharply reduce this memory burden.

In standard multi-head attention, each query head has its own dedicated key and value heads. GQA groups multiple query heads together to share a single set of key-value heads. This reduces memory requirements while preserving most of the representational capacity.

Understanding the attention variants

Before diving into the formulas, let's build intuition about the three attention variants:

1. Multi-Head Attention (MHA): The original design. Each query head gets its own private key-value pair. Maximum expressiveness, but maximum memory cost. Think of it as giving each employee their own filing cabinet.

2. Multi-Query Attention (MQA): The extreme efficiency approach. All query heads share a single key-value pair. Minimal memory, but information gets compressed. Like having everyone share one filing cabinet.

3. Grouped-Query Attention (GQA): The balanced solution. Small groups of query heads share key-value pairs. Qwen uses this because it retains most of MHA's quality while achieving most of MQA's efficiency gains.

The mathematical formulation

Now we can understand the formal definition. The overall GQA computation concatenates the outputs from all attention heads and projects them back to the model dimension:

$$\text{GQA}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$$

where:

• $Q, K, V$: the QKV tensors derived from the input hidden states
• $\text{head}_i$: the output of the $i$-th attention head, capturing what information token $i$ should attend to
• $h$: the total number of query heads (e.g., 32 for Qwen-7B, 64 for Qwen-72B)
• $W^O$: the output projection matrix of shape $(h \cdot d_{\text{head}}, d_{\text{model}})$ that recombines head outputs
• $\text{Concat}$: concatenation along the head dimension, stacking all head outputs together

Each individual head computes attention using its query projection but shares key-value projections with other heads in its group. This is where the efficiency gain comes from:

$$\text{head}_i = \text{Attention}(Q_i W^Q_i, K_{g(i)} W^K_{g(i)}, V_{g(i)} W^V_{g(i)})$$

where:

• $Q_i$: the query input for head $i$ (unique to each head)
• $W^Q_i$: the query projection matrix for head $i$ (unique to each head)
• $g(i)$: a grouping function that maps query head $i$ to its corresponding KV group index
• $K_{g(i)}, V_{g(i)}$: the key and value inputs for the group containing head $i$ (shared within group)
• $W^K_{g(i)}, W^V_{g(i)}$: the shared key and value projection matrices for that group

How grouping works in practice

The grouping function $g(i)$ determines which query heads share KV heads. For a model with $H_Q$ query heads and $H_{KV}$ key-value heads:

$$g(i) = \left\lfloor \frac{i \cdot H_{KV}}{H_Q} \right\rfloor$$

Let's trace through two concrete examples:

• Qwen-7B: 32 query heads, 32 KV heads (ratio 1:1). Here $g(i) = i$, so each query head has its own KV head. This is equivalent to standard MHA, the safe choice for a smaller model.

• Qwen-72B: 64 query heads, 8 KV heads (ratio 8:1). Every 8 consecutive query heads share the same KV head. Query heads 0-7 share KV head 0, heads 8-15 share KV head 1, and so on. This gives an 8× reduction in KV cache memory.

Why this works

The intuition for why grouping works is that different query heads often look for similar patterns in the keys and values. By forcing them to share, we impose a useful regularization. Empirically, researchers found that models with GQA ratios up to 8:1 show minimal quality degradation while achieving substantial memory and speed improvements.

Summary of advantages

• Reduced KV cache memory: Fewer KV heads means smaller cache during generation (8× reduction for Qwen-72B)
• Faster inference: Less memory bandwidth needed to load KV cache, directly improving tokens/second
• Maintained quality: Grouping preserves most of MHA's representational capacity through shared but still diverse key-value representations

Having understood how Qwen organizes its attention heads, we now turn to a subtler design choice: whether to include bias terms in the attention projections.

The design question

When projecting hidden states to QKV tensors, we have a choice:

$$\text{Without bias:} \quad Q = XW^Q \qquad \text{With bias:} \quad Q = XW^Q + b^Q$$

LLaMA follows the trend of removing all bias terms, arguing that the weight matrices alone have sufficient capacity and that removing bias simplifies the model. Qwen takes the opposite stance, adding bias to the QKV projections.

Why bias might help

The intuition is subtle but important. Bias terms give a learned "default" activation that doesn't depend on the input. In attention, this means:

• Queries can have a baseline direction they search in, independent of the current token
• Keys can have a baseline "signature" that makes certain patterns more or less likely to match
• Values can contribute baseline information even before considering specific content

Think of it like this: without bias, attention patterns must emerge entirely from token-token interactions. With bias, the model can learn that "queries from this head should generally attend more to earlier positions" or "values should include this baseline information regardless of content."

The parameter cost

This flexibility comes at minimal cost. For Qwen-7B:

$$\text{Bias parameters per layer} = 3 \times d_{\text{model}} = 3 \times 4096 = 12{,}288$$

Compared to the total attention parameters per layer (~67M), this is about 0.02%. Qwen's researchers found this negligible overhead gives measurable quality improvements, particularly in tasks requiring fine-grained attention patterns.

Implementation

Unlike LLaMA which removes all bias terms from linear projections, Qwen adds bias to the QKV projections:

The output shape matches the input shape, confirming the attention module correctly processes the sequence while preserving dimensions. The bias parameters add only 0.02% overhead to the attention layer, but Qwen's researchers found this small addition improves attention pattern learning. The 128-dimensional heads (4096 / 32) give sufficient capacity for complex attention patterns. While LLaMA removes biases for simplicity and to reduce parameters, Qwen found that the small parameter increase (less than 0.1% of total) gives real quality improvements.

After attention aggregates information across positions, the feed-forward network (FFN) processes each position independently, adding nonlinearity and increasing the model's representational capacity. Here we explore why Qwen chose SwiGLU over simpler alternatives.

The evolution of feed-forward design

The original Transformer used a simple two-layer FFN with ReLU activation:

$$\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$$

This works, but researchers discovered that gated variants perform better. The key insight: instead of applying a single nonlinearity, use one transformation to decide what information to pass through (the gate) and another to decide which information is available (the values).

The gating mechanism

SwiGLU introduces element-wise multiplication between two parallel branches:

$$\text{SwiGLU}(x) = \underbrace{\text{Swish}(xW_{\text{gate}})}_{\text{what to keep}} \odot \underbrace{(xW_{\text{up}})}_{\text{candidate values}}$$

Let's unpack each part:

1. Gate branch ($xW_{\text{gate}}$): Projects the input to an intermediate dimension, then applies Swish activation. The Swish output ranges from near-zero (suppress) to unbounded positive (amplify).

2. Value branch ($xW_{\text{up}}$): Projects the input to the same intermediate dimension. These are the "candidate" values that might pass through.

3. Element-wise product ($\odot$): For each element, the gate controls how much of the corresponding value passes through. Near-zero gate means that value is suppressed; large positive gate means that value is amplified.

The Swish activation

Why Swish rather than ReLU for the gate? Swish gives smooth, non-monotonic gating:

$$\text{Swish}(z) = z \cdot \sigma(z) = \frac{z}{1 + e^{-z}}$$

Key properties that make Swish effective:

• Smooth gradients: Unlike ReLU's hard cutoff at zero, Swish transitions gradually, improving gradient flow during training
• Non-monotonic: For small negative inputs, Swish outputs small negative values before approaching zero. This allows the gate to preserve subtle negative signals
• Self-gating: The $\sigma(z)$ term acts as a soft switch on the $z$ term itself

Analyzing the output

The MLP parameters dominate each transformer layer, accounting for roughly two-thirds of per-layer parameters. The 2.69x expansion ratio (rather than the standard 4x) compensates for SwiGLU's three projections instead of two, keeping total parameter count comparable to standard FFN layers while giving the gating mechanism's quality benefits.

Why gating matters

The key insight is that not all intermediate features are equally useful for every input. In a standard FFN, every intermediate neuron contributes to every output (modulated only by ReLU killing negative values). With SwiGLU, the gate branch learns when each intermediate feature should contribute, creating input-dependent computation paths. This is a form of learned sparsity that improves model quality without increasing inference cost.

Transformers process sequences in parallel, treating each position identically. This is computationally efficient but creates a problem: how does the model know that token 5 comes before token 10? The answer lies in positional encoding, and Qwen uses Rotary Position Embeddings (RoPE), the same approach as LLaMA.

The positional encoding challenge

Early transformers used absolute positional embeddings: learn a vector for position 0, another for position 1, and so on. This has two problems:

1. Length limitation: The model can only handle positions it saw during training
2. No relative reasoning: Positions 10-15 look completely different from positions 100-105, even though the relationship (5 tokens apart) is identical

What we really want is a way to encode position such that attention scores depend on relative distance between tokens, not their absolute positions.

The rotation insight

RoPE reaches this by encoding position as a rotation in embedding space. The key insight is that when two rotations are composed, the result depends only on their angular difference:

$$R_m^{-1} \cdot R_n = R_{n-m}$$

If we rotate queries by their position and keys by their position, then when we compute attention (which involves a dot product between queries and keys), the rotation effect depends only on the relative distance.

How rotation encodes position

For each position $m$, we construct a rotation matrix $R_m$ that rotates pairs of dimensions in the embedding space. The rotation angle depends on both the position and the dimension pair:

$$\theta_i = m \cdot \frac{1}{10000^{2i/d}}$$

where $i$ indexes the dimension pair (there are $d/2$ pairs for dimension $d$) and $m$ is the position. Lower-frequency rotations (larger $i$) encode longer-range position information; higher-frequency rotations (smaller $i$) encode fine-grained local position.