Weight Decay › Summary

Weight decay adds a penalty proportional to the squared magnitude of model weights to the training loss, discouraging large weights and encouraging solutions that generalize better. The key ideas from this chapter are:

• L2 regularization adds $\frac{\lambda}{2} \|\theta\|_2^2$ to the loss, producing a $(1 - \eta\lambda)$ shrinkage factor on weights at each update step. This is mathematically equivalent to weight decay for SGD but not for adaptive optimizers.
• Decoupled weight decay (AdamW) applies the weight decay term directly to parameters rather than through the loss gradient, making sure uniform regularization across all parameters regardless of their gradient history. This outperforms standard Adam with L2 regularization for most large-scale training setups.
• Selective application matters: bias terms and layer normalization parameters are typically excluded from weight decay to preserve training stability. The standard heuristic excludes all one-dimensional parameters.
• Coefficient selection typically starts around $\lambda = 0.01$ for transformer training and is tuned based on dataset size and model capacity, with larger values for smaller datasets. Published configurations from BERT ($0.01$), GPT-3 ($0.1$), and LLaMA ($0.1$) provide useful starting points.
• The learning rate coupling means effective regularization per step scales with both $\lambda$ and the learning rate, which is worth considering when using aggressive learning rate schedules.
• Fine-tuning requires care: standard weight decay pulls weights toward zero, not toward the pretrained initialization, which can hurt fine-tuning on small datasets. Use smaller $\lambda$ or L2-SP regularization when adapting pretrained models.

The next chapter covers gradient accumulation, another training optimization technique that addresses the practical constraint of GPU memory limits by simulating large batch training through accumulated gradient updates.