Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gated Linear Units › GeGLU: The GELU Variant

Michael BrenndoerferUpdated June 14, 20252 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 109Gated Linear Units › GeGLU: The GELU Variant

Reading section 6 of 16 · GeGLU: The GELU Variant · 2 minSection 6 of 16 · 2 min

Section 6 of 16 · 2 min read

Gated Linear Units › GeGLU: The GELU VariantOne section · Sections in this chapter6 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Another popular variant is GeGLU, which uses GELU instead of Swish for the gating function. The GeGLU formula follows the same pattern as SwiGLU, replacing the Swish activation with GELU:

\text{GeGLU}(x) = \text{GELU}(xW) \otimes (xV)

where:

$x \in \mathbb{R}^{d_{\text{model}}}$ : the input vector
$W \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ : the first projection matrix (GELU-activated pathway)
$V \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ : the second projection matrix (linear pathway)
$\otimes$ : element-wise multiplication (Hadamard product)

GELU (Gaussian Error Linear Unit) is defined as the product of the input and the cumulative distribution function (CDF) of the standard normal distribution:

\text{GELU}(z) = z \cdot \Phi(z)

where:

$z$ : the input value (a scalar, applied element-wise to vectors)
$\Phi(z)$ : the cumulative distribution function of the standard normal distribution, i.e., $\Phi(z) = P(Z \leq z)$ for $Z \sim \mathcal{N}(0, 1)$

Computing the exact CDF is expensive, so in practice a fast approximation is used:

\text{GELU}(z) \approx 0.5z \left(1 + \tanh\left[\sqrt{\frac{2}{\pi}}\left(z + 0.044715z^3\right)\right]\right)

This approximation uses the hyperbolic tangent function to approximate the error function that underlies the normal CDF. The constant $0.044715$ was empirically chosen to minimize approximation error across the typical input range.

GeGLU has similar properties to SwiGLU but with slightly different gradient characteristics. Some practitioners prefer GeGLU for encoder models (following BERT's use of GELU), while SwiGLU has become more common in decoder-only LLMs.

In[15]:

Code

def geglu(x, W, V):
    """
    GeGLU activation.

    Args:
        x: Input, shape (..., d_model)
        W: First projection weights, shape (d_model, d_ff)
        V: Second projection weights, shape (d_model, d_ff)

    Returns:
        Output, shape (..., d_ff)
    """
    return gelu(x @ W) * (x @ V)


def ffn_geglu(x, W, V, W2):
    """Feed-forward network with GeGLU activation."""
    hidden = geglu(x, W, V)
    return hidden @ W2


# Compare outputs of different GLU variants
y_geglu = ffn_geglu(x_single, W_swi, V_swi, W2_swi)

Out[16]:

Console

GLU variant comparison:
  SwiGLU output norm: 7.2403
  GeGLU output norm:  7.8467
  Difference norm:    1.2943

The output norms are similar between SwiGLU and GeGLU, indicating that both variants produce representations of comparable magnitude. The non-zero difference norm confirms that while the outputs are similar, they're not identical. These differences are subtle in terms of output statistics but can compound over many layers and matter for training dynamics and final model quality at scale.