Gated Linear Units › Key Parameters

When implementing or configuring GLU-based feed-forward networks:

• d_model (input/output dimension): The embedding dimension, unchanged from standard FFNs. Must match the attention layer output for proper residual connections.

• d_ff (hidden dimension): The dimension of the gated intermediate representation. For parameter parity with standard FFNs, use $d_{ff} = \frac{2}{3} \times 4 \times d_{\text{model}} = \frac{8}{3} \times d_{\text{model}}$. LLaMA rounds this to the nearest multiple of 256 for hardware efficiency. Higher ratios (3.5x or 4x) trade parameters for expressiveness.

• glu_variant: The choice of activation function for gating. SwiGLU ($\text{Swish}(xW) \otimes xV$) is the dominant choice. GeGLU uses GELU instead. ReGLU uses ReLU but is less common due to ReLU's sharp transitions.

• use_bias: Whether to include bias terms. Modern architectures (LLaMA, Mistral, PaLM) typically omit biases entirely, reducing parameters and simplifying quantization. The impact on model quality is minimal.

• multiple_of (LLaMA-specific): Round $d_{ff}$ to a multiple of this value (typically 256) for GPU memory alignment. The formula used is:

where $m$ is the multiple_of value and $\lceil \cdot \rceil$ denotes the ceiling function. For example, with $d_{\text{model}} = 4096$ and $m = 256$: the raw value is $\frac{8}{3} \times 4096 = 10922.67$, and rounding up to the nearest multiple of 256 gives $d_{ff} = 11008$.