AdamW › The AdamW Formulation

AdamW fixes the problem by applying weight decay directly to the parameters, separate from the gradient update. The regularization term never enters the moment estimates. The moments track only the gradient signal from the loss, and the weight decay operates independently as a direct multiplicative shrinkage of the parameters.

The standard Adam update with L2 regularization computes:

$$\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$

where the moments $m_t$ and $v_t$ are computed from the regularized gradient $g_t + \lambda \theta_t$.

The AdamW update decouples these components:

$$\theta_{t+1} = \theta_t - \alpha \left( \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda \theta_t \right)$$

where:

• $\hat{m}_t$: bias-corrected first moment, computed from the original gradient $g_t$ only
• $\hat{v}_t$: bias-corrected second moment, computed from the original gradient $g_t$ only
• $\lambda$: the weight decay coefficient
• $\alpha$: the learning rate

The weight decay term $\alpha \lambda \theta_t$ is added outside the adaptive scaling. Every parameter is shrunk by the same multiplicative factor $\alpha \lambda$ at each step, regardless of its gradient history. The adaptive learning rates only affect the gradient-based update; they have no influence on how strongly weight decay pulls each parameter toward zero.

Written out as the complete set of equations, AdamW performs these steps at each training iteration:

$$\begin{aligned} g_t &= \nabla_{\theta} L(\theta_{t-1}) \\ m_t &= \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ v_t &= \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 \\ \hat{m}_t &= \frac{m_t}{1 - \beta_1^t} \\ \hat{v}_t &= \frac{v_t}{1 - \beta_2^t} \\ \theta_t &= \theta_{t-1} - \alpha \left(\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda \theta_{t-1} \right) \end{aligned}$$

where:

• $g_t$: gradient of the training loss, with no regularization term mixed in
• $m_t$: first moment estimate, an exponential moving average of gradients
• $v_t$: second moment estimate, an exponential moving average of squared gradients
• $\hat{m}_t$: bias-corrected first moment, compensating for the zero initialization of $m_0$
• $\hat{v}_t$: bias-corrected second moment, compensating for the zero initialization of $v_0$
• $\beta_1$: first moment decay rate, typically 0.9
• $\beta_2$: second moment decay rate, typically 0.999
• $\epsilon$: numerical stability constant, typically 1e-8
• $\lambda$: weight decay coefficient
• $\alpha$: learning rate

The critical difference from Adam with L2 regularization is in the last equation. The weight decay applies uniformly: the same $\alpha \lambda \theta_{t-1}$ term acts on every parameter with the same effective coefficient, while the gradient update $\hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$ handles the adaptive per-parameter scaling independently.

Another way to think about the difference is geometric. Adam with L2 regularization is trying to minimize a particular objective function, namely the regularized loss $\tilde{L}(\theta)$, but using a preconditioned gradient descent step where the preconditioner is the adaptive scaling $1/\sqrt{\hat{v}_t}$. The preconditioning changes the effective geometry of the optimization landscape, and the L2 ball that L2 regularization defines in the original parameter space gets distorted by the preconditioner.

AdamW is doing something subtly different: it separates the geometry-distorting adaptive step for the loss gradient from the direct parameter shrinkage for weight decay. The adaptive step handles the optimization of the loss; the weight decay handles the regularization constraint. These two objectives are kept separate rather than being mixed into a single gradient computation.

This is also why AdamW has a cleaner connection to the original weight decay formulation from SGD literature. In the SGD with momentum world, weight decay was a direct multiplicative shrinkage applied to parameters. AdamW restores this interpretation in the adaptive optimizer setting, where Adam's adaptive scaling makes it impossible to achieve uniform weight decay through gradient modification alone.