AdamW › L2 Regularization vs. Weight Decay

Understanding the distinction between L2 regularization and weight decay is the core insight behind AdamW. These two approaches address the same goal from different angles, and for a long time, practitioners used the terms interchangeably. They are identical in the context of standard gradient descent, but they diverge for adaptive optimizers, and that divergence has significant practical consequences.

L2 regularization adds a penalty term to the training loss. The modified objective function becomes:

$$\tilde{L}(\theta) = L(\theta) + \frac{\lambda}{2} \|\theta\|^2$$

where:

• $L(\theta)$: the original training loss (cross-entropy, MSE, etc.)
• $\lambda$: the regularization coefficient controlling penalty strength
• $\|\theta\|^2 = \sum_i \theta_i^2$: the squared L2 norm of all parameters
• $\tilde{L}(\theta)$: the regularized loss that the optimizer minimizes

When you compute the gradient of $\tilde{L}$ with respect to a parameter $\theta_i$, the regularization term contributes an extra additive term:

$$\nabla_{\theta_i} \tilde{L}(\theta) = \nabla_{\theta_i} L(\theta) + \lambda \theta_i$$

The gradient now has two components: the original loss gradient, which points toward lower training loss, plus $\lambda \theta_i$, which always pushes $\theta_i$ toward zero. The optimizer sees a single combined gradient, with no distinction between the signal from data and the signal from regularization. The optimizer does not know that part of the gradient it receives is a regularization term rather than a data-derived signal. It simply processes the combined gradient as if it were all signal from the loss.

Weight decay is a different approach to the same goal. Instead of modifying the loss function, weight decay modifies the parameter update directly. After computing the gradient update from the loss, you shrink the weights by a fixed fraction:

$$\theta_{t+1} = (1 - \lambda) \theta_t - \alpha \nabla_{\theta} L(\theta_t)$$

where:

• $\theta_t$: the parameter value at step $t$
• $\lambda$: the weight decay coefficient
• $\alpha$: the learning rate
• $\nabla_{\theta} L(\theta_t)$: the gradient of the training loss

The weight decay term $(1-\lambda)\theta_t$ multiplies the current parameter value by a number slightly less than 1, which shrinks it toward zero. This shrinkage is completely independent of the gradient. It happens as a direct multiplicative operation on the parameter value, regardless of what the gradient is doing or how large recent gradients have been. The gradient step and the weight decay step are conceptually separate operations, even though they are combined into a single update equation.

For standard stochastic gradient descent with learning rate $\alpha$, the two approaches produce mathematically identical updates. The SGD update with L2 regularization is:

$$\begin{aligned} \theta_{t+1} &= \theta_t - \alpha \nabla_{\theta} \tilde{L}(\theta_t) \\ &= \theta_t - \alpha (\nabla_{\theta} L(\theta_t) + \lambda \theta_t) \\ &= (1 - \alpha \lambda) \theta_t - \alpha \nabla_{\theta} L(\theta_t) \end{aligned}$$

Compare this to the weight decay update: $\theta_{t+1} = (1 - \lambda') \theta_t - \alpha \nabla_{\theta} L(\theta_t)$.

These are identical when $\lambda' = \alpha \lambda$. You can translate between the two by rescaling the regularization coefficient by the learning rate. For SGD, L2 regularization and weight decay are equivalent formulations of the same thing, just parameterized differently.

This equivalence is why many practitioners and many codebases used the terms interchangeably. Weight decay in PyTorch's SGD optimizer actually implements L2 regularization via the gradient, and for SGD it makes no difference. The naming inconsistency in the deep learning ecosystem is a direct consequence of this historical equivalence being assumed to hold universally, when in fact it breaks down as soon as you move beyond SGD.

Even though L2 regularization and weight decay are equivalent for SGD, they reflect different computational philosophies. L2 regularization thinks about regularization as part of the objective: you are literally optimizing a different function, one that includes a preference for small weights. Weight decay thinks about regularization as part of the update rule: you are adding a direct shrinkage operation that is applied after the gradient step.

For simple first-order optimizers, this distinction does not matter because the gradient step scales uniformly across all parameters. But once you introduce any form of adaptive scaling, the two approaches diverge because the adaptive scaling modifies the gradient-based update but not the direct parameter update.