Weight Decay › A Worked Example: Weight Decay Step by Step

To make the math concrete, let us trace through a single weight decay update for a scalar weight.

Suppose we have a single weight $\theta = 2.5$, a task loss gradient at this point of $\frac{\partial L}{\partial \theta} = -0.8$, a learning rate $\eta = 0.01$, and a weight decay coefficient $\lambda = 0.1$.

Without weight decay, the standard SGD update is:

$$\begin{aligned} \theta &\leftarrow \theta - \eta \cdot \frac{\partial L}{\partial \theta} \\ &= 2.5 - 0.01 \times (-0.8) \\ &= 2.5 + 0.008 \\ &= 2.508 \end{aligned}$$

The weight increases from 2.5 to 2.508 because the gradient is negative (the loss decreases as the weight increases).

With weight decay, the update adds the regularization gradient $\lambda \theta = 0.1 \times 2.5 = 0.25$:

$$\begin{aligned} \theta &\leftarrow \theta - \eta \left(\frac{\partial L}{\partial \theta} + \lambda \theta\right) \\ &= 2.5 - 0.01 \times (-0.8 + 0.1 \times 2.5) \\ &= 2.5 - 0.01 \times (-0.8 + 0.25) \\ &= 2.5 - 0.01 \times (-0.55) \\ &= 2.5 + 0.0055 \\ &= 2.5055 \end{aligned}$$

The weight still increases, but by less. The task loss gradient wanted to push the weight up by 0.008, but the regularization gradient is opposing this by adding a positive component (0.0025) to the total gradient update in the direction of shrinking the weight.

Equivalently, using the shrinkage form:

$$\begin{aligned} \theta &\leftarrow (1 - \eta \lambda)\,\theta - \eta \frac{\partial L}{\partial \theta} \\ &= (1 - 0.01 \times 0.1) \times 2.5 - 0.01 \times (-0.8) \\ &= 0.999 \times 2.5 + 0.008 \\ &= 2.4975 + 0.008 \\ &= 2.5055 \end{aligned}$$

The two forms give the same result, as expected. The shrinkage form makes visible that the weight is first multiplied by 0.999 (a 0.1% reduction) and then adjusted by the task gradient. Over 1000 steps with no gradient contribution, the weight would decay from 2.5 toward zero following a geometric series: $2.5 \times 0.999^{1000} \approx 2.5 \times 0.368 \approx 0.92$.

This exponential decay toward zero is where the name "weight decay" originates. At every step, the optimizer applies a small multiplicative shrinkage to each weight, continuously counteracting the tendency for weights to grow large. The task gradient can push weights away from zero when a feature improves the predictions, but it must constantly overcome the decay pressure to do so.

There is an instructive way to think about what weight decay does to the steady-state weight magnitude. Suppose a weight has converged and is no longer changing on average. At that point, the average contribution of the task gradient must exactly cancel the contribution of the weight decay term:

$$\mathbb{E}\left[\frac{\partial L}{\partial \theta}\right] = -\lambda \theta$$

This means the steady-state magnitude of the weight is determined by the ratio of the expected task gradient to the weight decay coefficient:

$$|\theta^*| \approx \frac{1}{\lambda} \left| \mathbb{E}\left[\frac{\partial L}{\partial \theta}\right] \right|$$

A stronger weight decay (larger $\lambda$) forces weights to be smaller in equilibrium, which is only possible if the gradient signal for that weight is also stronger. Parameters that contribute little to reducing the task loss (small gradients) will be pushed toward zero. Parameters that contribute strongly (large gradients for important features) will stabilize at larger values. Weight decay thus implements a soft version of importance weighting: only weights that are useful enough to overcome the decay pressure remain large.

This equilibrium perspective also clarifies why weight decay cannot cause catastrophic collapse. If the weight decay were too aggressive and drove all weights toward zero, the model would make random predictions, generating large gradients that push weights back up. The equilibrium is stable: the task gradient always provides a restoring force proportional to how poorly the model performs. Weight decay shifts the equilibrium point but cannot collapse the model to zero as long as the task has a learnable signal.

To see why large weights are the mechanism behind overfitting (rather than just a symptom), consider what happens during unconstrained gradient descent on a small dataset. The optimizer follows a path of steepest descent in the loss landscape, always moving in the direction that most rapidly reduces training loss. Early in training, this is fine: both training and validation loss decrease as the model learns recurring patterns. But as the model approaches the training data manifold, the loss surface develops sharp minima that correspond to memorizing individual training examples.

These sharp minima are characterized by large weights. The model learns to respond with high confidence to specific training examples by amplifying the weights connecting the relevant input features to the output. This amplification is invisible to the optimizer, which sees only that training loss continues to decrease. But you can see the consequence in the weight norms: they grow throughout training as the model memorizes more and more.

Weight decay changes the shape of the loss surface. The penalty term adds a bowl-shaped component to the total loss, converting sharp minima into smoother, broader minima. The optimizer can no longer rest in narrow memorization minima because the cost of staying there (the penalty for large weights) exceeds the benefit (the reduction in training loss). Instead, it settles in broader minima that correspond to learned patterns rather than memorized examples. These broader minima are precisely the solutions that generalize well to unseen data.