Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gated Linear Units › GLU in the Feed-Forward Network

Michael BrenndoerferUpdated June 14, 20253 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 109Gated Linear Units › GLU in the Feed-Forward Network

Reading section 4 of 16 · GLU in the Feed-Forward Network · 3 minSection 4 of 16 · 3 min

Section 4 of 16 · 3 min read

Gated Linear Units › GLU in the Feed-Forward NetworkOne section · Sections in this chapter4 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

With the GLU mechanism understood, our next step is integrating it into the transformer's feed-forward network. This requires adapting the standard two-layer FFN architecture, and understanding this integration reveals why GLU is such a natural fit for transformers.

Recalling the Standard FFNLink Copied

The standard FFN applies a simple pattern: expand the dimension, apply nonlinearity, contract back:

\text{FFN}(x) = \sigma(xW_1 + b_1)W_2 + b_2

where $x$ is the input, $W_1$ and $W_2$ are weight matrices, $b_1$ and $b_2$ are bias vectors, and $\sigma$ is a nonlinear activation function (e.g., ReLU or GELU).

The key insight is that the nonlinearity $\sigma$ is applied after the expansion. This creates a high-dimensional feature space where the nonlinearity can carve out complex decision boundaries, before projecting back to the original dimension.

Replacing Activation with GatingLink Copied

GLU replaces the simple activation function with something more sophisticated: a learned gating mechanism. Instead of uniformly transforming all features with the same nonlinearity, GLU lets the network selectively amplify or suppress features based on the input itself:

\text{FFN}_{\text{GLU}}(x) = \left[(xW + b) \otimes \sigma(xV + c)\right] W_2 + b_2

Let's trace through what each component does in this context:

Expansion to Hidden Space: Both $xW$ and $xV$ project the input from $d_{\text{model}}$ to the larger hidden dimension $d_{ff}$
Gated Combination: The sigmoid-gated multiplication produces a $d_{ff}$ -dimensional hidden representation
Contraction Back: $W_2$ projects back to $d_{\text{model}}$ for the residual connection

The variables are:

$x \in \mathbb{R}^{d_{\text{model}}}$ : input from attention/normalization
$W, V \in \mathbb{R}^{d_{\text{model}} \times d_{ff}}$ : the two projection matrices for value and gate
$b, c \in \mathbb{R}^{d_{ff}}$ : corresponding bias vectors
$W_2 \in \mathbb{R}^{d_{ff} \times d_{\text{model}}}$ : output projection back to model dimension
$b_2 \in \mathbb{R}^{d_{\text{model}}}$ : output bias

The Parameter CostLink Copied

Notice that the GLU FFN requires three weight matrices ( $W$ , $V$ , and $W_2$ ) instead of the standard FFN's two ( $W_1$ and $W_2$ ). This means GLU-based FFNs have 50% more parameters for the same hidden dimension. We'll analyze this trade-off in detail later and see how modern models address it.

In[8]:

Code

def ffn_glu(x, W, b, V, c, W2, b2):
    """
    Feed-forward network with GLU activation.

    Args:
        x: Input, shape (..., d_model)
        W, b: Value projection (d_model -> d_ff)
        V, c: Gate projection (d_model -> d_ff)
        W2, b2: Output projection (d_ff -> d_model)

    Returns:
        Output, shape (..., d_model)
    """
    # GLU combines value and gate
    value = x @ W + b
    gate = sigmoid(x @ V + c)
    hidden = value * gate

    # Project back to model dimension
    output = hidden @ W2 + b2
    return output


# FFN dimensions
d_model_ffn = 512
d_ff = 2048

# Initialize all weights
W_ffn = np.random.randn(d_model_ffn, d_ff) * np.sqrt(2.0 / (d_model_ffn + d_ff))
b_ffn = np.zeros(d_ff)
V_ffn = np.random.randn(d_model_ffn, d_ff) * np.sqrt(2.0 / (d_model_ffn + d_ff))
c_ffn = np.zeros(d_ff)
W2_ffn = np.random.randn(d_ff, d_model_ffn) * np.sqrt(
    2.0 / (d_ff + d_model_ffn)
)
b2_ffn = np.zeros(d_model_ffn)

# Process a sequence
seq_len = 16
X = np.random.randn(seq_len, d_model_ffn)
Y = ffn_glu(X, W_ffn, b_ffn, V_ffn, c_ffn, W2_ffn, b2_ffn)

Out[9]:

Console

GLU FFN example:
  d_model: 512
  d_ff:    2048
  Sequence length: 16
  Input shape:  (16, 512)
  Output shape: (16, 512)

The output shape matches the input shape, confirming that the GLU FFN correctly projects back to the model dimension. This dimensional consistency is essential for the residual connection that adds the FFN output back to its input in a transformer block.