Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LLaMA Components › The SwiGLU Feed-Forward Network

Michael BrenndoerferUpdated August 2, 20259 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 208LLaMA Components › The SwiGLU Feed-Forward Network

Reading section 3 of 12 · The SwiGLU Feed-Forward Network · 9 minSection 3 of 12 · 9 min

Section 3 of 12 · 9 min read

LLaMA Components › The SwiGLU Feed-Forward NetworkOne section · Sections in this chapter3 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The feed-forward network (FFN) in each transformer block provides the model's primary source of nonlinearity and parameter capacity. While attention handles communication between tokens, the FFN handles computation within each token's representation. This is where the model stores and applies learned knowledge, from factual information to linguistic patterns.

LLaMA uses SwiGLU, a gated variant that has become the standard in modern language models. To understand why SwiGLU works better than simpler alternatives, we'll trace the evolution from basic FFNs to gated architectures, seeing how each improvement addresses specific limitations.

From Standard FFN to Gated UnitsLink Copied

The original transformer FFN follows a simple recipe: project each token to a higher dimension, apply a nonlinearity, then project back. This "expand and contract" pattern allows the network to compute complex functions of the input through the nonlinearity in the high-dimensional space:

\text{FFN}(\mathbf{x}) = \text{ReLU}(\mathbf{x}\mathbf{W}_1)\mathbf{W}_2

where:

$\mathbf{x} \in \mathbb{R}^d$ : input token representation with dimension $d$
$\mathbf{W}_1 \in \mathbb{R}^{d \times d_{\text{ff}}}$ : first projection, expanding to hidden dimension $d_{\text{ff}}$ (typically $4d$ )
$\mathbf{W}_2 \in \mathbb{R}^{d_{\text{ff}} \times d}$ : second projection, compressing back to model dimension
$\text{ReLU}(z) = \max(0, z)$ : rectified linear unit activation

This architecture works, but it has a limitation: every hidden dimension is processed identically by the ReLU. The network cannot selectively emphasize some features while suppressing others based on the input content.

Gated linear units (GLUs) address this limitation by introducing a learned control mechanism. Instead of applying a fixed activation to all hidden dimensions, a GLU learns which dimensions to activate and which to suppress. It does this by splitting the hidden layer into two parallel paths: one computes a "gate" signal that controls information flow, while the other computes the actual content. These paths are multiplied element-wise:

\text{GLU}(\mathbf{x}) = (\mathbf{x}\mathbf{W}_1 \odot \sigma(\mathbf{x}\mathbf{V}))\mathbf{W}_2

where:

$\mathbf{x} \in \mathbb{R}^d$ : input token representation
$\mathbf{W}_1 \in \mathbb{R}^{d \times d_{\text{ff}}}$ : linear path projection
$\mathbf{V} \in \mathbb{R}^{d \times d_{\text{ff}}}$ : gate path projection
$\mathbf{W}_2 \in \mathbb{R}^{d_{\text{ff}} \times d}$ : output projection
$\sigma(z) = 1/(1 + e^{-z})$ : sigmoid activation function
$\odot$ : element-wise (Hadamard) multiplication

The sigmoid-gated path acts as a learned filter, controlling how much of the linear path's information passes through. When $\sigma(\mathbf{x}\mathbf{V})$ is near zero for some hidden dimension, that dimension's contribution is blocked; when near one, it passes through fully.

SwiGLU: Swish-Gated Linear UnitsLink Copied

While the sigmoid-gated GLU works well, researchers found an even better gating function: Swish (also called SiLU). The Swish function has an elegant property: it is "self-gating," meaning the input gates itself. Rather than having a separate sigmoid control the gate, Swish multiplies the input by its own sigmoid, creating a smooth, adaptive activation.

SwiGLU applies this insight to the GLU architecture:

\text{SwiGLU}(\mathbf{x}) = (\text{Swish}(\mathbf{x}\mathbf{W}_{\text{gate}}) \odot (\mathbf{x}\mathbf{W}_{\text{up}}))\mathbf{W}_{\text{down}}

where:

$\mathbf{x} \in \mathbb{R}^d$ : input token representation with model dimension $d$
$\mathbf{W}_{\text{gate}} \in \mathbb{R}^{d \times d_{\text{ff}}}$ : gate projection matrix
$\mathbf{W}_{\text{up}} \in \mathbb{R}^{d \times d_{\text{ff}}}$ : up projection matrix (the "linear path")
$\mathbf{W}_{\text{down}} \in \mathbb{R}^{d_{\text{ff}} \times d}$ : down projection, mapping back to model dimension
$\odot$ : element-wise multiplication between gate and up projections

The Swish activation function is defined as:

\text{Swish}(z) = z \cdot \sigma(z) = \frac{z}{1 + e^{-z}}

where $\sigma(z)$ is the sigmoid function. Swish is smooth everywhere (unlike ReLU's sharp corner at zero) and self-gated: the input $z$ is multiplied by its own sigmoid, meaning large positive values pass through nearly unchanged while negative values are attenuated. This combination has been shown to improve training dynamics and final model quality compared to both ReLU and GELU-based FFNs.

Out[7]:

Visualization

Line plot comparing ReLU, GELU, and Swish activation functions, showing their different behaviors for positive and negative inputs. — Comparison of activation functions used in transformer FFNs. ReLU has a hard cutoff at zero; GELU provides smooth gating; Swish combines self-gating with smooth gradients.

Line plot comparing gradients of ReLU, GELU, and Swish, showing that Swish and GELU maintain non-zero gradients throughout. — Gradient (derivative) of each activation function. ReLU has zero gradient for negative inputs (dead neurons). GELU and Swish maintain non-zero gradients throughout, improving training.

Parameter Count Trade-off

SwiGLU requires three weight matrices ( $\mathbf{W}_{\text{gate}}$ , $\mathbf{W}_{\text{up}}$ , $\mathbf{W}_{\text{down}}$ ) instead of two ( $\mathbf{W}_1$ , $\mathbf{W}_2$ ). To keep the total parameter count similar, LLaMA reduces the hidden dimension from $4d$ to approximately $\frac{8d}{3} \approx 2.67d$ . The exact value is often rounded to be divisible by hardware-friendly numbers.

Out[8]:

Visualization

Bar chart comparing parameter counts between Standard FFN and SwiGLU, showing how the reduced hidden dimension compensates for the extra matrix. — FFN parameter comparison: Standard FFN (2 matrices with 4d hidden) vs SwiGLU (3 matrices with 2.67d hidden). Despite using three matrices, SwiGLU achieves similar parameter count by reducing the hidden dimension.

With the theory in place, let's implement SwiGLU and see how the gating mechanism works in practice. We'll first define the Swish activation, then build the complete SwiGLU module:

In[9]:

Code

def swish(x):
    """Swish (SiLU) activation: x * sigmoid(x)"""
    return x / (1 + np.exp(-x))


class SwiGLU:
    """
    SwiGLU feed-forward network as used in LLaMA.

    Implements: FFN(x) = (Swish(x @ W_gate) * (x @ W_up)) @ W_down
    """

    def __init__(self, d_model, d_ff):
        """
        Args:
            d_model: Model dimension
            d_ff: Hidden dimension (typically ~2.67 * d_model for SwiGLU)
        """
        self.d_model = d_model
        self.d_ff = d_ff

        # Xavier/Glorot initialization
        scale_up = np.sqrt(2.0 / (d_model + d_ff))
        scale_down = np.sqrt(2.0 / (d_ff + d_model))

        self.W_gate = np.random.randn(d_model, d_ff) * scale_up
        self.W_up = np.random.randn(d_model, d_ff) * scale_up
        self.W_down = np.random.randn(d_ff, d_model) * scale_down

    def __call__(self, x):
        """Apply SwiGLU transformation."""
        gate = swish(x @ self.W_gate)  # Gating signal
        up = x @ self.W_up  # Linear projection
        hidden = gate * up  # Element-wise gating
        return hidden @ self.W_down  # Project back to d_model

Out[10]:

Visualization

Line plot showing Swish activation compared to ReLU, with Swish giving smooth gradients throughout. — Swish compared to ReLU activation. Swish has smooth non-zero gradients for negative inputs, unlike ReLU's hard cutoff at zero.

Line plot showing how gating modulates information flow in SwiGLU. — SwiGLU gating mechanism. The gate signal modulates information flow, blocking when negative and letting passage when positive.

The left plot shows why Swish is preferred over ReLU. Unlike ReLU's hard cutoff at zero, Swish has a smooth curve with non-zero gradients for negative inputs. This helps gradient flow during training, particularly for inputs that happen to fall in negative regions.

The right plot illustrates gating. When the gate input is negative, the gate output is near zero, blocking information. As the gate input increases, more of the up projection passes through. This learned gating allows the network to selectively process different aspects of the input.