AdamW › Summary

AdamW corrects a subtle but consequential flaw in how standard Adam handles regularization. The key ideas are:

• L2 regularization and weight decay are equivalent for SGD, because SGD applies a uniform learning rate to all parameters. The regularization term in the gradient gets the same scaling as the loss gradient, so the two formulations produce identical parameter updates up to a rescaling of the coefficient.
• They diverge for adaptive optimizers like Adam, because Adam's per-parameter adaptive scaling applies different effective learning rates to different parameters. When the regularization penalty is mixed into the gradient, it gets scaled differently for each parameter based on that parameter's gradient history. Parameters with large historical gradients receive weaker effective regularization than parameters with small historical gradients.
• In language models, this matters most for embeddings, where common token embeddings receive dense, large gradients and thus weak effective regularization under Adam+L2, while rare token embeddings receive sparse, small gradients and thus strong effective regularization. This is backwards: common tokens, updated most often, are the most at risk of overfitting.
• AdamW fixes this by decoupling: the weight decay is applied directly to the parameters as $\alpha \lambda \theta_t$, outside the adaptive scaling term. Every parameter receives the same weight decay factor at each step, and the regularization coefficient $\lambda$ has a consistent, interpretable meaning independent of gradient history.
• The hyperparameter semantics change: a weight_decay value in AdamW behaves predictably, while the same value in standard Adam does not. Optimal weight decay values for AdamW tend to be larger than what was used with Adam+L2 because the effective regularization per parameter is more stable.
• Standard practice in LLM training uses AdamW with weight_decay=0.01 to 0.1, learning rate 1e-4 to 3e-4 with warmup and cosine decay, and bias terms plus LayerNorm parameters excluded from weight decay.
• AdamW is the default optimizer for LLM training: GPT-2, GPT-3, BERT, and essentially all major language models since 2018 use AdamW, and it remains the dominant choice for large-scale training as of 2025.

The next chapter covers weight initialization, the practice of setting parameter starting values before training begins. Poor initialization can cause gradients to vanish or explode even before the first update, making choices like Xavier and He initialization essential for training deep networks reliably.