Transformer Block Assembly › Key Parameters

When implementing transformer blocks, these parameters control capacity and efficiency:

• $d_{\text{model}}$ (model dimension): The embedding dimension that flows through the entire block. Typical values range from 256 (small models) to 12288 (GPT-3 scale). This dimension must be divisible by $n_{\text{heads}}$ since each head operates on $d_{\text{head}} = d_{\text{model}} / n_{\text{heads}}$ dimensions.

• $n_{\text{heads}}$ (attention heads): Number of parallel attention mechanisms. More heads enable attending to different types of relationships simultaneously. Common choices are 8, 12, 16, or 32 heads.

• $d_{\text{ff}}$ (FFN hidden dimension): The intermediate dimension of the feed-forward network. For standard FFNs, typically $d_{\text{ff}} = 4 \times d_{\text{model}}$. For SwiGLU, typically $d_{\text{ff}} = \frac{8}{3} \times d_{\text{model}}$ (approximately 2.7×) to maintain similar parameter count. The gating mechanism effectively doubles the width, so a smaller explicit expansion compensates.

• norm_type: Choice between RMSNorm (faster, used in LLaMA/Mistral) and LayerNorm (original, used in GPT-2/BERT). RMSNorm is preferred for new architectures.

• norm_placement: Pre-norm (normalize before sublayers) or post-norm (normalize after). Pre-norm is standard for modern deep models due to better gradient flow.

• ffn_type: Standard (two linear layers with activation) or gated (SwiGLU, GeGLU). Gated variants improve expressiveness at similar parameter cost.

• causal: Whether to apply causal masking in attention. Required for autoregressive language models, disabled for bidirectional encoders like BERT.

• $\epsilon$ (normalization epsilon): Small constant (typically $10^{-6}$ to $10^{-5}$) added to prevent division by zero in normalization. The RMSNorm formula divides by $\sqrt{\text{mean}(x^2) + \epsilon}$. Without $\epsilon$, a vector of all zeros would cause a division by zero. May need adjustment (e.g., $10^{-5}$ instead of $10^{-6}$) for mixed-precision training where numerical precision is lower.