Gated Linear Units › Summary

Gated Linear Units transform the standard feed-forward network by introducing multiplicative interactions through learned gates. Rather than applying a simple activation function, GLU computes two parallel projections and multiplies them together, with one pathway controlling how much of the other passes through.

Key takeaways:

• The gating mechanism: GLU computes $(xW) \otimes \sigma(xV)$, where $x$ is the input, $W$ and $V$ are learned projection matrices, $\sigma$ is the sigmoid function producing gate values in $[0, 1]$, and $\otimes$ denotes element-wise multiplication. This multiplicative interaction enables richer representations than additive transformations alone, capturing feature interactions that standard activations cannot express.

• SwiGLU dominance: Modern LLMs predominantly use SwiGLU, defined as $\text{Swish}(xW) \otimes (xV)$ where $\text{Swish}(z) = z \cdot \sigma(z)$ is a self-gating activation that multiplies the input by its own sigmoid. The self-gating property of Swish, combined with the linear pathway, provides smooth gradients and unbounded positive outputs.

• GeGLU alternative: GeGLU replaces Swish with GELU. This gives similar benefits with slightly different gradient characteristics. It's sometimes preferred for encoder models following BERT's GELU tradition.

• Parameter trade-offs: GLU requires three weight matrices ($W$, $V$, $W_2$) instead of two, adding 50% more parameters for the same hidden dimension. To maintain parameter parity, models reduce $d_{ff}$ to approximately $\frac{2}{3}$ of the standard value. LLaMA uses $d_{ff} = \frac{8}{3} \times d_{\text{model}}$ instead of the standard $4 \times d_{\text{model}}$.

• Universal adoption: Since 2022, virtually every state-of-the-art LLM uses SwiGLU or a similar gated variant. LLaMA and Mistral use gated activations; PaLM and related models do as well. This shows consistent improvements in perplexity and later tasks.

• Smoother activations: Unlike ReLU's hard sparsity, SwiGLU produces smoother activation distributions with preserved negative values. This improves gradient flow during training and allows the network to preserve graded feature information.

The next chapter assembles the components we've covered into complete transformer blocks. We'll see how the ordering and connections among residual and normalization components, plus attention and feed-forward components, create the architecture that underlies modern language models.