Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gated Linear Units › SwiGLU: The Modern Standard

Michael BrenndoerferUpdated June 14, 20259 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 109Gated Linear Units › SwiGLU: The Modern Standard

Reading section 5 of 16 · SwiGLU: The Modern Standard · 9 minSection 5 of 16 · 9 min

Section 5 of 16 · 9 min read

Gated Linear Units › SwiGLU: The Modern StandardOne section · Sections in this chapter5 / 16 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The original GLU with sigmoid gating works, but researchers discovered an even better formulation. SwiGLU, introduced by Shazeer in 2020, has become the dominant choice across modern LLMs, including frontier models such as LLaMA and Mistral. Understanding why SwiGLU outperforms the original GLU reveals important insights about activation function design.

The Limitation of Sigmoid GatingLink Copied

Recall that the original GLU uses sigmoid to produce gate values between 0 and 1. While this works well for controlling information flow, sigmoid has a subtle limitation: it saturates at both ends. For very large positive inputs, the gate is always ~1; for very large negative inputs, always ~0. This saturation can limit expressiveness and cause gradient flow issues in deep networks.

From Sigmoid to Swish: A Self-Gating ActivationLink Copied

Swish (also called SiLU for Sigmoid Linear Unit) offers an elegant solution. Instead of using sigmoid as a separate gate, Swish embeds gating directly into the activation function:

\text{Swish}(z) = z \cdot \sigma(z)

where $z$ is the input value and $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid function.

This formula is deceptively simple but powerful. Let's understand what it does:

For large positive $z$ : $\sigma(z) \approx 1$ , so $\text{Swish}(z) \approx z$ (passes through like a linear function)
For large negative $z$ : $\sigma(z) \approx 0$ , so $\text{Swish}(z) \approx 0$ (suppressed like ReLU)
For small $z$ near zero: both $z$ and $\sigma(z)$ contribute, creating a smooth transition

The key insight is that Swish is self-gating: the input determines its own gate value. This creates an implicit "soft attention" where each value decides how much of itself to pass through.

The SwiGLU FormulaLink Copied

SwiGLU combines Swish with the two-pathway GLU structure:

\text{SwiGLU}(x) = \text{Swish}(xW) \otimes (xV)

where:

$x \in \mathbb{R}^{d_{\text{model}}}$ : the input vector
$W \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ : the first projection matrix (Swish-activated pathway)
$V \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ : the second projection matrix (linear pathway)
$\otimes$ : element-wise multiplication (Hadamard product)

Expanding Swish reveals the full structure:

\text{SwiGLU}(x) = (xW \cdot \sigma(xW)) \otimes (xV)

The Asymmetry ExplainedLink Copied

Notice something different from the original GLU: the asymmetry. In SwiGLU:

The first pathway ( $xW$ ) passes through Swish, which includes self-gating
The second pathway ( $xV$ ) remains completely linear

This differs from the original GLU where sigmoid was applied specifically to the "gate" pathway and the "value" pathway was linear. In SwiGLU, the Swish-activated pathway acts as a kind of self-gated value, while the linear pathway provides additional expressiveness through the multiplication.

The combination creates a rich interaction: you have two different views of the input (through $W$ and $V$ ), one filtered through self-gating nonlinearity, combined multiplicatively. This gives the network more degrees of freedom than either a simple activation or the original GLU.

Why SwiGLU Outperforms the OriginalLink Copied

Several factors contribute to SwiGLU's success:

Unbounded positive values: Unlike sigmoid (which saturates at 1), Swish can output arbitrarily large positive values, matching the range of ReLU-family activations
Smooth gradients: Swish is differentiable everywhere with smooth gradients, avoiding the sharp transitions at zero that ReLU has
Self-gating property: The formula $z \cdot \sigma(z)$ means each dimension gates itself, creating input-dependent behavior without explicit gate parameters
Better gradient flow: The linear term in Swish ensures gradients can flow even for extreme inputs, helping train deep networks

The gradient behavior is particularly important for training deep networks. Let's visualize how gradients flow differently through each activation:

Out[10]:

Visualization

Line plot comparing gradients of ReLU, GELU, Swish, and sigmoid, showing how Swish maintains gradients for negative inputs. — Gradient (derivative) of different activation functions. ReLU has zero gradient for negative inputs ('dead neurons' problem), while Swish and GELU maintain small but non-zero gradients that allow information to flow during backpropagation.

ImplementationLink Copied

Let's implement SwiGLU and see these properties in action:

In[11]:

Code

def swish(x):
    """Swish (SiLU) activation: x * sigmoid(x)."""
    return x * sigmoid(x)


def swiglu(x, W, V):
    """
    SwiGLU activation (biases often omitted in modern implementations).

    Args:
        x: Input, shape (..., d_model)
        W: First projection weights, shape (d_model, d_ff)
        V: Second projection weights, shape (d_model, d_ff)

    Returns:
        Output, shape (..., d_ff)
    """
    return swish(x @ W) * (x @ V)


def ffn_swiglu(x, W, V, W2):
    """
    Feed-forward network with SwiGLU activation.

    Modern implementations typically omit biases.
    """
    hidden = swiglu(x, W, V)
    return hidden @ W2


# Initialize weights for SwiGLU FFN
W_swi = np.random.randn(d_model_ffn, d_ff) * np.sqrt(2.0 / (d_model_ffn + d_ff))
V_swi = np.random.randn(d_model_ffn, d_ff) * np.sqrt(2.0 / (d_model_ffn + d_ff))
W2_swi = np.random.randn(d_ff, d_model_ffn) * np.sqrt(
    2.0 / (d_ff + d_model_ffn)
)

# Test
x_single = np.random.randn(d_model_ffn)
y_swiglu = ffn_swiglu(x_single, W_swi, V_swi, W2_swi)

Out[12]:

Console

SwiGLU FFN example:
  Input shape:  (512,)
  Output shape: (512,)
  Output norm:  7.2403

The output norm provides a rough measure of the magnitude of the transformation. With Xavier initialization, we expect the output norm to be similar to the input norm, indicating stable signal propagation through the network.

Let's examine the self-gating behavior of Swish more closely by looking at how the implicit gate values (the sigmoid component) distribute for random inputs:

Out[13]:

Visualization

Histogram showing distribution of Swish gate values with 50% suppression threshold marked. — Distribution of implicit gate values in Swish activation. Values below 0.5 indicate suppression, while values above allow signal to pass through.

Scatter plot of Swish input vs output colored by gate value, showing smooth transition. — Swish input-output relationship showing smooth transition from suppression (negative inputs) to linear pass-through (positive inputs).

Let's also visualize how Swish compares to other activation functions and how this affects the gating behavior:

Out[14]:

Visualization

Line plot comparing ReLU, GELU, Swish, and sigmoid activation functions. — Comparison of activation functions. Swish (used in SwiGLU) provides a smooth, unbounded alternative to ReLU with a small negative region that can help gradient flow.

Line plot showing gating characteristics of different activations. — The effective gating behavior of each activation when used in a GLU-style multiplication. Swish provides smoother gating than sigmoid while allowing values greater than 1.