AdamW › A Concrete Numerical Comparison

To see the difference concretely, consider two parameters in a language model:

• Parameter A: an embedding for a common word, frequently updated with large gradients. Suppose $\hat{v}_t = 0.01$ (reflecting a large gradient history).
• Parameter B: an embedding for a rare word, infrequently updated with small gradients. Suppose $\hat{v}_t = 0.0001$ (reflecting a small gradient history).

With L2 regularization in Adam (coupling), the effective regularization multiplier for each parameter is roughly $\lambda / (\sqrt{\hat{v}_t} + \epsilon)$. Setting $\lambda = 0.01$ and $\epsilon = 10^{-8}$:

• Parameter A (common word): effective regularization $\approx 0.01 / \sqrt{0.01} = 0.01 / 0.1 = 0.1$
• Parameter B (rare word): effective regularization $\approx 0.01 / \sqrt{0.0001} = 0.01 / 0.01 = 1.0$

Parameter B experiences ten times stronger effective regularization than parameter A, simply because it has a smaller gradient history. But consider what this means in practice. The common word embedding has been updated on nearly every training step and has had the most opportunity to overfit to training data. The rare word embedding has been updated infrequently and is likely underdetermined, not overfitted. Adam+L2 applies strong regularization to the underdetermined parameter and weak regularization to the potentially overfitted one: exactly backwards from what you would want.

With AdamW (decoupled weight decay), both parameters receive the same $\alpha \lambda = \alpha \times 0.01$ decay factor at every step. Parameter A and parameter B are both shrunk by the same multiplicative factor, regardless of how often they have been updated or how large their historical gradients have been. The regularization is uniform, as intended.

This concrete example illustrates why AdamW matters particularly for language models, which have exactly this structure: a vocabulary of many tokens with wildly varying frequencies, producing embedding gradients with highly heterogeneous statistics.