Gated Linear Units › GLU in Modern Architectures

Gated Linear Units have become the de facto standard in modern large language models. Here's how different architectures implement GLU variants:

• LLaMA (Meta, 2023): Uses SwiGLU with $d_{ff} = \frac{8}{3} \times d_{\text{model}}$ rounded to the nearest multiple of 256 for hardware efficiency. The rounding ensures tensor dimensions align well with GPU architectures. Biases are completely removed from the FFN.

• Mistral (2023): Also uses SwiGLU but with a higher ratio ($d_{ff} \approx 3.5 \times d_{\text{model}}$), trading parameters for expressiveness. Like LLaMA, no biases are used.

• PaLM (Google, 2022): Uses SwiGLU with a 4x expansion ratio, accepting the higher parameter count in exchange for expressiveness. The model uses separate embedding and softmax matrices and applies RMS normalization.

• Falcon (TII, 2023): Uses GELU rather than Swish in its GLU variant, following the BERT tradition while still benefiting from the gating mechanism.