Weight Decay › The Weight Decay Penalty

To understand weight decay, start with the loss function. During training you minimize some loss $L(\theta)$ that measures how poorly your model's predictions match the training targets, where $\theta$ represents all the model parameters. Gradient descent updates parameters to reduce this loss. Without any constraint, the optimizer is free to set weights to any value it wants, including extremely large ones that perfectly fit the training data.

This unconstrained optimization is the root of the problem. Think of a model with 1 million parameters and only 1,000 training examples. The training loss can be driven to zero (or near zero) by a solution that memorizes every training example, but this solution will perform no better than chance on new data. The model has more than enough degrees of freedom to interpolate through every training point exactly, which is precisely what an unconstrained optimizer will eventually find if you train long enough. The key insight behind weight decay is that memorizing solutions tend to use large weights: the model must respond sharply to specific input patterns to correctly classify individual training examples, and sharp responses require large weight magnitudes.

Weight decay adds a regularization term directly to the loss:

$$L_{\text{total}}(\theta) = L(\theta) + \frac{\lambda}{2} \|\theta\|_2^2$$

where:

• $L(\theta)$: the original task loss (cross-entropy, MSE, etc.)
• $\lambda$: the weight decay coefficient, a positive hyperparameter controlling regularization strength
• $\|\theta\|_2^2$: the squared L2 norm of all parameters, defined as $\sum_i \theta_i^2$
• The factor $\frac{1}{2}$ is a mathematical convenience that cancels during differentiation

The gradient of this combined loss with respect to any parameter $\theta_i$ is:

$$\frac{\partial L_{\text{total}}}{\partial \theta_i} = \frac{\partial L}{\partial \theta_i} + \lambda \theta_i$$

where:

• $\frac{\partial L}{\partial \theta_i}$: the gradient of the task loss with respect to parameter $i$
• $\lambda \theta_i$: the regularization gradient, which is simply the parameter value scaled by $\lambda$

When you apply gradient descent with learning rate $\eta$, the update rule becomes:

$$\theta_i \leftarrow \theta_i - \eta \left(\frac{\partial L}{\partial \theta_i} + \lambda \theta_i\right)$$

Expanding this:

$$\theta_i \leftarrow (1 - \eta \lambda)\,\theta_i - \eta \frac{\partial L}{\partial \theta_i}$$

This formulation is revealing. The $(1 - \eta \lambda)$ factor multiplies the parameter before subtracting the gradient. Since $\eta \lambda$ is a small positive number (typically something like $0.01 \times 0.0001 = 10^{-6}$), you are shrinking the parameter slightly toward zero at every step before moving in the gradient direction. This is why weight decay is sometimes called L2 regularization and why it produces the "weight shrinkage" effect that gives the technique its name.

The shrinkage effect is cumulative. After $T$ steps with no gradient contribution, a weight $\theta$ would decay to:

$$\theta_T = (1 - \eta \lambda)^T \cdot \theta_0$$

For typical values of $\eta = 10^{-3}$ and $\lambda = 10^{-2}$, we have $\eta \lambda = 10^{-5}$, so after 10,000 steps without any gradient signal, a weight would retain $(1 - 10^{-5})^{10000} \approx e^{-0.1} \approx 0.905$ of its original magnitude. After 100,000 steps, it would retain about $e^{-1} \approx 0.37$. This slow but relentless decay is what keeps unused features from persisting indefinitely. Any weight that the task gradient does not consistently reinforce will eventually shrink toward zero.

The intuition behind penalizing large weights connects to the concept of function smoothness. A function implemented by a neural network with large weights responds strongly to small changes in input features. A small perturbation in the input can produce a large change in the output. This extreme sensitivity to inputs is the hallmark of a memorized solution: the model has learned to respond intensely to specific patterns in the training data, which may not generalize.

Consider a simple linear model $f(x) = w \cdot x + b$. If $w$ is very large, then a tiny change in $x$ produces a huge change in $f(x)$. The model is fragile. With L2 regularization, the optimizer must weigh the benefit of increasing $w$ to reduce training loss against the cost of increasing the $w^2$ penalty. This creates pressure toward smaller weights, which correspond to smoother, more robust functions.

There is also a probabilistic interpretation. Minimizing the L2-regularized loss is equivalent to maximum a posteriori (MAP) estimation with a Gaussian prior centered at zero on the weights. The prior expresses a belief that weights are likely to be small, and the regularization strength $\lambda$ controls how strongly you hold that prior belief relative to the evidence from the training data. Bayesian practitioners often think about weight decay in exactly these terms.

The geometry of this constraint is worth visualizing. In two-dimensional weight space, the L2 penalty defines concentric circular contours centered at the origin. The task loss defines its own contours centered at the unconstrained optimum. The regularized solution is found where the total loss surface, combining both, has its minimum. This generally pulls the solution toward the origin compared to the unregularized minimum.

This geometric view shows something important: the regularized solution is almost never exactly at the origin. The task gradient must be exactly zero for weight decay alone to drive a weight to zero, which happens only for completely useless parameters. For all parameters that contribute something to the task, the optimal regularized weight is a compromise between what the task demands and what the penalty allows. This is the key difference between L2 regularization and constraints: L2 regularization does not enforce a hard constraint on weight magnitude, it merely penalizes large magnitudes. The model retains the freedom to grow weights when the task gradient justifies the cost.

The connection between weight decay and Bayesian inference is more than a curiosity. It provides a rigorous justification for why weight decay is appropriate as a default regularizer and gives you a principled way to think about the strength of the prior.

In the Bayesian framing, you place a prior distribution $p(\theta)$ over model parameters before observing any data. A Gaussian prior centered at zero, $p(\theta_i) \propto \exp(-\theta_i^2 / (2\sigma^2))$, expresses the belief that weights are probably small and symmetric around zero. The maximum a posteriori (MAP) estimate is then:

$$\theta^* = \arg\max_\theta \; p(\theta \mid \mathcal{D}) = \arg\max_\theta \; \log p(\mathcal{D} \mid \theta) + \log p(\theta)$$

Substituting the Gaussian prior:

$$\theta^* = \arg\max_\theta \; \log p(\mathcal{D} \mid \theta) - \frac{1}{2\sigma^2} \sum_i \theta_i^2$$

Negating to turn this into a minimization:

$$\theta^* = \arg\min_\theta \; -\log p(\mathcal{D} \mid \theta) + \frac{1}{2\sigma^2} \sum_i \theta_i^2$$

The first term is the negative log-likelihood (the task loss), and the second term is exactly the L2 regularization penalty with $\lambda = 1/\sigma^2$. A small $\sigma^2$ (tight prior) corresponds to large $\lambda$ (strong regularization). A large $\sigma^2$ (diffuse prior) corresponds to small $\lambda$ (weak regularization). When $\sigma^2 \to \infty$, the prior becomes uninformative and weight decay disappears.

This probabilistic interpretation suggests a principled way to think about choosing $\lambda$: it reflects your prior belief about how large model weights should be. If you believe the task is well-suited to the model architecture and the data is representative, you might use a diffuse prior (small $\lambda$). If you are working with a model that is much larger than the training data warrants, a tighter prior (larger $\lambda$) is appropriate. This matches the empirical conventions described earlier: larger models relative to dataset size call for larger $\lambda$.

Weight decay uses the squared L2 norm, but a natural question is: why not the L1 norm instead? L1 regularization adds $\lambda \sum_i |\theta_i|$ to the loss, which has a fundamentally different effect on the learned parameters.

The gradient of the L1 penalty with respect to $\theta_i$ is $\lambda \cdot \text{sign}(\theta_i)$, a constant magnitude regardless of how large or small $\theta_i$ is. This means L1 regularization applies the same push toward zero whether a weight is 0.001 or 1000. The consequence is that L1 tends to produce exactly-zero weights for parameters that do not contribute to reducing the task loss, leading to sparse solutions.

L2 regularization, by contrast, has a gradient of $\lambda \theta_i$ that is proportional to the parameter's current value. Large weights receive strong push-back; small weights receive little. This gradient decreases as weights approach zero, which is why L2 shrinks weights toward zero without setting them exactly to zero. The result is dense solutions where most weights remain nonzero but are collectively kept small.

For neural network training, L2 (weight decay) is almost always preferred over L1. There are three main reasons. First, L2 is smooth and differentiable everywhere, making gradient-based optimization straightforward. L1 has a non-differentiable kink at zero, requiring subgradient methods or proximal operators. Second, dense solutions with uniformly small weights generalize well for neural networks, where many parameters contribute small amounts to the final prediction. Third, L1's sparsity-inducing property is less useful in neural networks than in linear models, since neural networks already have architectural mechanisms (neurons, attention heads) that provide a more interpretable form of functional sparsity.

There are settings where L1 is useful: when you want to prune weights explicitly, when working with linear models for feature selection, or when you want to induce embedding sparsity. But for end-to-end deep learning, L2 weight decay is the standard choice.

There is also a probabilistic analogy here. While L2 regularization corresponds to a Gaussian prior (smooth, symmetric), L1 regularization corresponds to a Laplace prior (heavier tails, sharper peak at zero). The Laplace prior places more probability mass exactly at zero, which is why L1 tends to produce exact zeros. The Gaussian prior places more probability mass at small but nonzero values, which is why L2 produces small but nonzero weights. For most neural network applications, the Gaussian prior is a better model of what we believe about the weight distribution.