Pre-Norm vs Post-Norm › Modern Architectures: The Pre-Norm Consensus

The empirical evidence strongly favors pre-norm for training stability, especially in large models. Let's examine how major architectures make this choice.

The transition from post-norm to pre-norm was not a sudden paradigm shift but a gradual convergence driven by engineering necessity. BERT (2018) used post-norm and 12 layers, a depth at which post-norm's instabilities are manageable with careful warmup. GPT-2 (2019) adopted pre-norm and pushed to 48 layers in its largest variant; OpenAI engineers noted that training was markedly more stable than post-norm would have allowed at that scale. T5 (2019) from Google independently converged on pre-norm for its 24-layer configurations. By the time LLaMA was released in 2023 with 32 layers and RMSNorm as a pre-norm variant, pre-norm had become so entrenched that choosing post-norm for a new large model would require special justification.

Notice that BERT and RoBERTa, along with DistilBERT, all use post-norm and are still among the most widely fine-tuned models in production. This is because fine-tuning is a much gentler process than pre-training from scratch. Starting from a pre-trained checkpoint means the model is already near a good optimum; you are taking small gradient steps from a stable starting point rather than fighting instability during thousands of warmup steps. When practitioners encounter BERT in deployment, post-norm is not a liability. It only becomes one when training from scratch at significant depth.

The pattern is clear: newer and larger models overwhelmingly choose pre-norm. The stability benefits outweigh any theoretical concerns about activation accumulation.