Weight Decay › Decoupled Weight Decay and AdamW

Standard weight decay works beautifully with stochastic gradient descent. The math lines up: adding the L2 penalty to the loss is exactly equivalent to decaying the weights by a fixed fraction at each step. However, when you move to adaptive gradient methods like Adam, this equivalence breaks down in a way that significantly matters in practice.

Before examining the problem, it helps to recall how Adam updates parameters. Adam maintains per-parameter adaptive learning rates. For each parameter $\theta_i$, it tracks the first moment (exponential moving average of gradients) $m_i$ and the second moment (exponential moving average of squared gradients) $v_i$. The full update procedure at step $t$ is:

$$\begin{aligned} m_i &\leftarrow \beta_1 m_i + (1 - \beta_1) g_i \\ v_i &\leftarrow \beta_2 v_i + (1 - \beta_2) g_i^2 \\ \hat{m}_i &\leftarrow \frac{m_i}{1 - \beta_1^t} \\ \hat{v}_i &\leftarrow \frac{v_i}{1 - \beta_2^t} \\ \theta_i &\leftarrow \theta_i - \eta \frac{\hat{m}_i}{\sqrt{\hat{v}_i} + \epsilon} \end{aligned}$$

where:

• $g_i$: the gradient of the loss with respect to parameter $i$
• $\beta_1$: decay rate for the first moment (typically 0.9)
• $\beta_2$: decay rate for the second moment (typically 0.999)
• $\hat{m}_i$: bias-corrected first moment estimate, correcting for initialization at zero
• $\hat{v}_i$: bias-corrected second moment estimate
• $\epsilon$: small constant for numerical stability (typically $10^{-8}$)

The key insight about Adam is that the denominator $\sqrt{\hat{v}_i} + \epsilon$ normalizes the update by the root mean square of past gradients. Parameters that have historically large gradients receive smaller updates (the large denominator divides them down), while parameters with historically small gradients receive larger updates. This per-parameter rescaling is what makes Adam so effective for neural networks, where different parameters may have very different natural gradient scales.

When you add L2 regularization to the loss, the gradient passed to Adam becomes $g_i + \lambda \theta_i$, where $g_i$ is the task loss gradient. This regularization gradient gets folded into both the first and second moment estimates:

$$\begin{aligned} m_i &\leftarrow \beta_1 m_i + (1 - \beta_1)(g_i + \lambda \theta_i) \\ v_i &\leftarrow \beta_2 v_i + (1 - \beta_2)(g_i + \lambda \theta_i)^2 \end{aligned}$$

As a result, the adaptive learning rate scale, which is computed from gradient history, is influenced by the regularization term. The second moment $v_i$ now reflects the history of both the task gradient and the weight decay gradient.

The consequence is that the effective weight decay depends on the history of gradients for that parameter. Parameters with small gradients from the task loss will have small $v_i$ values. Adam would normally amplify their updates (small denominator). But because $\lambda \theta_i$ is also folded into $v_i$, the second moment estimate is inflated by the regularization term, partially suppressing this amplification. The actual regularization strength varies in complicated ways across parameters, defeating the purpose of having a clean $\lambda$ hyperparameter.

The deeper problem is this: the adaptive scaling that Adam applies to regularize the step size based on gradient history should apply to the task signal, not to the weight decay signal. Weight decay is supposed to be a global shrinkage applied uniformly to all parameters. When it gets folded into Adam's adaptive mechanism, the global uniformity is lost. Parameters with small task gradients receive weaker regularization, which is the opposite of what you want: these are often the parameters most likely to drift to unusual values, since Adam amplifies their updates.

Ilya Loshchilov and Frank Hutter identified this problem in their 2017 paper "Decoupled Weight Decay Regularization" and proposed a fix: separate the weight decay step from the gradient update step.

AdamW modifies the Adam update by applying weight decay directly to the parameters, independently of the adaptive gradient scaling. The update rule becomes:

$$\begin{aligned} m_i &\leftarrow \beta_1 m_i + (1 - \beta_1) g_i \\ v_i &\leftarrow \beta_2 v_i + (1 - \beta_2) g_i^2 \\ \hat{m}_i &\leftarrow m_i / (1 - \beta_1^t) \\ \hat{v}_i &\leftarrow v_i / (1 - \beta_2^t) \\ \theta_i &\leftarrow \theta_i - \eta \left(\frac{\hat{m}_i}{\sqrt{\hat{v}_i} + \epsilon} + \lambda \theta_i\right) \end{aligned}$$

where:

• $g_i$: the gradient of the task loss (not including the regularization term)
• $\lambda \theta_i$: the weight decay term applied directly to the parameter, outside the adaptive scaling
• All other terms are as in standard Adam

The critical difference is in the last line. The weight decay term $\lambda \theta_i$ is added to the update outside the adaptive denominator $\sqrt{\hat{v}_i} + \epsilon$. This means the weight decay applies uniformly to all parameters at the same rate $\lambda$, regardless of their gradient history. The adaptive learning rate only affects the gradient term, not the regularization term.

Expanding the AdamW update:

$$\theta_i \leftarrow (1 - \eta \lambda)\,\theta_i - \eta \frac{\hat{m}_i}{\sqrt{\hat{v}_i} + \epsilon}$$

This is analogous to the SGD case: multiply the parameter by $(1 - \eta \lambda)$ to shrink it, then subtract the adaptively scaled gradient. The weight decay effect is clean and uniform.

In practice, AdamW consistently outperforms Adam with L2 regularization for training large language models and other deep networks. The reason is that uniform weight decay across all parameters is a more coherent regularization strategy. When Adam applies L2 regularization internally, parameters with small gradients (often those that represent rarely-used features or less-frequently updated neurons) receive weaker regularization. These are exactly the parameters most at risk of drifting to unusual values, since the optimizer's adaptive scaling amplifies their updates. AdamW avoids this asymmetry.

The practical difference is especially visible when you tune the weight decay hyperparameter. With standard Adam + L2, the optimal $\lambda$ tends to be entangled with the learning rate in a complex way, making hyperparameter search difficult. AdamW decouples these, making $\lambda$ and $\eta$ more independently tunable and simplifying the optimization landscape.

A concrete way to see the difference: suppose you have a parameter $\theta_i$ with consistently small task gradients, so $v_i$ is small (Adam will amplify its updates). With L2 Adam, the regularization term $\lambda \theta_i$ gets divided by the small $\sqrt{v_i}$, making the effective regularization on this parameter much larger than $\lambda$ would suggest. With AdamW, the regularization is exactly $\eta \lambda \theta_i$ regardless of $v_i$. This predictability is what makes AdamW a reliable optimizer for large-scale training.

To put numbers on this: if $\sqrt{v_i} = 0.001$ (a rarely-updated parameter), $\eta = 0.0001$, $\lambda = 0.01$, and $\theta_i = 1.0$, then:

• With L2 Adam: effective weight decay per step is approximately $\eta \lambda \theta_i / \sqrt{v_i} = 0.0001 \times 0.01 \times 1.0 / 0.001 = 0.001$ (100 times stronger than intended)
• With AdamW: effective weight decay per step is $\eta \lambda \theta_i = 0.0001 \times 0.01 \times 1.0 = 0.000001$ (exactly as intended)

The L2 Adam approach accidentally applies extremely strong regularization to rarely-updated parameters, which is exactly backwards from what a well-calibrated regularizer should do. AdamW keeps regularization proportional to $\lambda$ throughout, regardless of the parameter's gradient history.

It might seem surprising that a flaw in how L2 regularization interacts with Adam went unnoticed for several years after Adam was introduced in 2014. The reason is that for many tasks, the difference between L2 Adam and AdamW is modest: both apply some form of regularization, and both work better than no regularization at all. The difference becomes most visible when you are carefully tuning hyperparameters for large-scale pretraining, where the precise meaning of the weight decay coefficient matters and the scale of training amplifies small differences. The natural language processing community discovered the practical importance of AdamW through large-scale BERT and GPT training, where the cleaner regularization semantics of AdamW translated into measurable improvements.

The Loshchilov and Hutter paper also documented a related issue: the optimal learning rate for Adam with L2 regularization depends on the weight decay coefficient, because both affect the effective step size. This coupling makes grid search over hyperparameters much harder. AdamW breaks this coupling, making hyperparameter search over $\eta$ and $\lambda$ more independent.