Pre-Norm vs Post-Norm › Key Parameters

When implementing pre-norm or post-norm transformer blocks, these parameters control the behavior and stability:

• norm_type: Choice between "pre" or "post". Pre-norm places LayerNorm before the sublayer; post-norm places it after the residual addition. Modern large models (GPT, LLaMA) use pre-norm for stability.

• eps (LayerNorm epsilon): Small constant (typically 1e-5 or 1e-6) added to the variance for numerical stability. Prevents division by zero when variance is very small.

• gamma, beta (LayerNorm parameters): Learnable scale and shift parameters. Initialized to ones and zeros respectively, allowing the model to learn the optimal normalization behavior for each layer.

• residual_scale: For pre-norm networks, scaling the output of sublayers by a factor like $1/\sqrt{2n}$ (where $n$ is the number of layers) prevents activation explosion. GPT-2 uses this technique.

• n_layers: Network depth directly impacts the choice between pre-norm and post-norm. Networks deeper than 12 layers benefit significantly from pre-norm's gradient stability.

• learning_rate: Post-norm requires smaller learning rates and warmup schedules. Pre-norm tolerates larger learning rates (often 3-10x higher) and simpler schedules.