Architecture Hyperparameters › Feed-Forward Network Dimensions

While attention layers decide which tokens to combine, the feed-forward network (FFN) performs the actual transformation of each token's representation. This division of labor has a surprising consequence: the FFN contains far more parameters than attention, making it the dominant factor in model size.

Understanding why requires examining what each component does. Attention computes weighted averages of value vectors, a fundamentally linear operation (the only nonlinearity is softmax for computing weights). The FFN, by contrast, applies a nonlinear transformation that can reshape the representation space, activate or suppress features, and encode factual associations. Research has shown that specific FFN neurons activate for particular concepts: "Eiffel Tower," "Python programming," or "past tense verbs." The FFN is where the model stores much of its learned knowledge.

This knowledge-storage role explains why FFNs need large capacity. The original transformer paper established the convention $d_{\text{ff}} = 4 \times d_{\text{model}}$. This provides the FFN with four times more hidden dimensions than the model's representation size. This 4x expansion gives each layer a wide intermediate space for computation before projecting back to $d_{\text{model}}$ dimensions.

Let's derive exactly how this expansion affects parameter counts and understand why FFNs dominate model size.

For a standard two-layer FFN, the computation applies an up-projection and activation before the down-projection:

$$\text{FFN}(\mathbf{x}) = \text{GELU}(\mathbf{x} \mathbf{W}_1 + \mathbf{b}_1) \mathbf{W}_2 + \mathbf{b}_2$$

where:

• $\mathbf{x} \in \mathbb{R}^{d_{\text{model}}}$: the input vector (one token's representation)
• $\mathbf{W}_1 \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$: the up-projection weight matrix
• $\mathbf{W}_2 \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$: the down-projection weight matrix
• $\mathbf{b}_1 \in \mathbb{R}^{d_{\text{ff}}}$, $\mathbf{b}_2 \in \mathbb{R}^{d_{\text{model}}}$: bias vectors
• $\text{GELU}(\cdot)$: the Gaussian Error Linear Unit activation function

The total parameter count includes both weight matrices and bias vectors:

$$\text{FFN parameters} = 2 \times d_{\text{model}} \times d_{\text{ff}} + d_{\text{ff}} + d_{\text{model}}$$

The first term accounts for the two weight matrices (each with $d_{\text{model}} \times d_{\text{ff}}$ parameters), while the remaining terms count the bias vectors. With the standard expansion ratio $d_{\text{ff}} = 4 \times d_{\text{model}}$ and ignoring the relatively small bias terms:

$$\text{FFN parameters} \approx 2 \times d_{\text{model}} \times 4 \times d_{\text{model}} = 8 \times d_{\text{model}}^2$$

Compare this to attention parameters for standard multi-head attention (MHA), which has four projection matrices (QKV and output), each of size $d_{\text{model}} \times d_{\text{model}}$:

$$\text{Attention parameters} = 4 \times d_{\text{model}}^2$$

The FFN has twice as many parameters as attention. This 2:1 ratio is consistent across most transformer architectures and explains why FFN optimization is so important for efficiency.

The visualization confirms that regardless of architecture choice, FFN parameters dominate. Even with SwiGLU's reduced expansion ratio, the FFN still accounts for roughly twice the parameters of attention. This consistent 2:1 ratio has practical consequences for optimization techniques like mixture-of-experts, which specifically target FFN layers.

The standard FFN applies a simple pattern: project up, apply nonlinearity, project down. But research has shown that adding a gating mechanism can improve performance significantly. The gating idea comes from LSTMs and GRUs: instead of passing all information through uniformly, learn which dimensions to emphasize and which to suppress.

SwiGLU (SiLU-Gated Linear Unit) implements this by computing two parallel projections of the input. One projection becomes the "content" that might pass through. The other projection, after applying the SiLU activation, becomes the "gate" that controls how much of each dimension passes. The element-wise product of content and gate creates a selective filtering effect.

This gating mechanism requires three weight matrices instead of two:

$$\text{SwiGLU}(\mathbf{x}) = (\text{SiLU}(\mathbf{x}\mathbf{W}_{\text{gate}}) \odot \mathbf{x}\mathbf{W}_{\text{up}}) \mathbf{W}_{\text{down}}$$

where:

• $\mathbf{x} \in \mathbb{R}^{d_{\text{model}}}$: the input vector
• $\mathbf{W}_{\text{gate}} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$: the gate projection matrix
• $\mathbf{W}_{\text{up}} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ff}}}$: the up-projection matrix
• $\mathbf{W}_{\text{down}} \in \mathbb{R}^{d_{\text{ff}} \times d_{\text{model}}}$: the down-projection matrix
• $\text{SiLU}(\cdot)$: the Sigmoid Linear Unit activation (also called Swish)
• $\odot$: element-wise multiplication

With three matrices each containing $d_{\text{model}} \times d_{\text{ff}}$ parameters, the total count is:

$$\text{SwiGLU parameters} = 3 \times d_{\text{model}} \times d_{\text{ff}}$$

To maintain the same parameter count as a standard FFN with 4x expansion, we can solve for the required SwiGLU expansion ratio. Setting the SwiGLU parameter count equal to the standard FFN count:

$$3 \times d_{\text{model}} \times d_{\text{ff}}^{\text{SwiGLU}} = 2 \times d_{\text{model}} \times 4 \times d_{\text{model}}$$

Dividing both sides by $3 \times d_{\text{model}}$ gives the required FFN dimension:

$$d_{\text{ff}}^{\text{SwiGLU}} = \frac{8}{3} \times d_{\text{model}} \approx 2.67 \times d_{\text{model}}$$

In practice, $d_{\text{ff}}$ is often rounded to a multiple of 256 for efficient GPU computation. LLaMA uses $d_{\text{ff}} = 2.6875 \times d_{\text{model}}$ (specifically 11008 for $d_{\text{model}} = 4096$), which is close to the theoretical value of $\frac{8}{3} \times 4096 = 10923$.