Gated Linear Units › The GLU Formulation

Now that we understand what gating does, let's formalize how it works mathematically. The Gated Linear Unit was introduced by Dauphin et al. in 2017 for language modeling, and its elegance lies in a simple but powerful insight: instead of applying an activation function to transform features, let the network learn which features to keep and which to suppress.

Standard neural network layers compute a single transformation of their input and apply an activation function. GLU takes a fundamentally different approach: it computes two separate linear projections of the same input and combines them through multiplication. One projection produces the "values" we want to transmit, while the other produces "gates" that control how much of each value passes through.

Think of it like a mixing console in a recording studio. Each channel has both a signal (the audio content) and a fader (the volume control). The final output for each channel is the signal multiplied by the fader position. The GLU works the same way, but learns both the signals and the fader positions from data.

The original GLU formula captures this two-pathway structure:

$$\text{GLU}(x) = (xW + b) \otimes \sigma(xV + c)$$

Let's unpack each component to understand what it contributes:

The Value Pathway: $(xW + b)$

The first term computes a linear transformation of the input. This is the "content" that we might want to pass through to the next layer. Without the gate, this would just be a standard linear layer:

• $x \in \mathbb{R}^{d_{\text{model}}}$: the input vector (a token's representation)
• $W \in \mathbb{R}^{d_{\text{model}} \times d}$: the "value" projection weights that transform the input
• $b \in \mathbb{R}^{d}$: the value projection bias

The Gate Pathway: $\sigma(xV + c)$

The second term computes another linear transformation of the same input, but passes it through the sigmoid function to produce gate values:

• $V \in \mathbb{R}^{d_{\text{model}} \times d}$: the "gate" projection weights (separate from $W$)
• $c \in \mathbb{R}^{d}$: the gate projection bias
• $\sigma$: the sigmoid activation function, which squashes values to the range $(0, 1)$
• $d$: the output dimension of each projection

The Combination: $\otimes$ (Element-wise Multiplication)

The $\otimes$ symbol denotes the Hadamard product, which simply multiplies corresponding elements. If the value pathway produces $[v_1, v_2, v_3]$ and the gate pathway produces $[g_1, g_2, g_3]$, the output is $[v_1 \cdot g_1, v_2 \cdot g_2, v_3 \cdot g_3]$.

The sigmoid function produces values between 0 and 1, making it a natural choice for gating:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

where $z$ is the input value (applied element-wise when the input is a vector). The sigmoid's behavior at extreme values is what makes it effective as a gate:

• As $z \to -\infty$, $\sigma(z) \to 0$ (gate closes completely)
• As $z \to +\infty$, $\sigma(z) \to 1$ (gate opens fully)
• At $z = 0$, $\sigma(z) = 0.5$ (gate is half-open)

When $\sigma(xV + c)$ is close to 0, the corresponding element of $xW + b$ is blocked, meaning that dimension contributes nothing to the output. When it's close to 1, the element passes through unchanged. Values between 0 and 1 allow partial transmission, giving the network fine-grained control over information flow.

Let's implement the basic GLU and see these components in action:

The output range shows that GLU produces both positive and negative values, unlike ReLU which only produces non-negative outputs. This is because the sigmoid gate modulates but doesn't constrain the sign of the value pathway.

Notice something important about GLU's output dimension. If we project from $d_{\text{model}}$ to $d$, we get $d$ outputs, not $2d$. This is because we use two projections (value and gate), each producing $d$ dimensions, and multiply them together element-wise to get $d$ outputs. The multiplication combines the two pathways rather than concatenating them.

This has clear implications for parameter efficiency that we'll explore shortly. First, let's see how GLU fits into the larger transformer architecture.