Weight Decay: L2 Regularization, AdamW, Decoupled Training

Neural networks have an enormous number of parameters. A modern large language model can have hundreds of billions of them, but even a modest classifier trained on a small dataset might have millions. With that many degrees of freedom, the network can memorize training data rather than learn to generalize. It fits noise, quirks of the specific examples it saw, and spurious patterns that do not hold in the real world. This failure mode is called overfitting, and combating it is one of the central challenges in training.

Weight decay is one of the oldest and most effective tools for preventing overfitting. The idea is simple: penalize large weights. A model with large weights is a model that depends heavily on specific input features, which is often a sign of memorization rather than generalization. By adding a penalty term to the loss function that grows with the magnitude of the weights, you encourage the optimizer to prefer simpler, smoother solutions that are more likely to transfer to unseen data.

Weight decay sits alongside other regularization techniques like dropout, which was covered in Part X. While dropout works by randomly zeroing activations during training, weight decay operates directly on the loss function, pushing parameter values toward zero throughout training. The two techniques are complementary and are often used together. Their combination addresses two different sources of overfitting: dropout prevents co-adaptation of neurons, while weight decay prevents any individual weight from growing large enough to dominate the model's behavior.

As we covered in the chapters on learning rate warmup, learning rate decay, cosine schedules, and large batch training, training optimization involves many interacting decisions. Weight decay is another lever in this system, one that is particularly important when working with large models on limited data, or when using optimizers like Adam where the interaction between weight decay and adaptive gradient scaling requires special care. Getting weight decay right can mean the difference between a model that generalizes cleanly and one that memorizes training quirks.

This chapter develops weight decay from first principles: why large weights are problematic, how the L2 penalty addresses this at both the loss-function level and the parameter update level, why standard L2 regularization fails to work as intended with adaptive optimizers like Adam, how AdamW fixes this with decoupled weight decay, and how to apply these ideas correctly in practice. By the end, you will understand how to set a weight decay coefficient, why you are setting it, and what it does to the model's learning dynamics.

To understand weight decay, start with the loss function. During training you minimize some loss $L(\theta)$ that measures how poorly your model's predictions match the training targets, where $\theta$ represents all the model parameters. Gradient descent updates parameters to reduce this loss. Without any constraint, the optimizer is free to set weights to any value it wants, including extremely large ones that perfectly fit the training data.

This unconstrained optimization is the root of the problem. Think of a model with 1 million parameters and only 1,000 training examples. The training loss can be driven to zero (or near zero) by a solution that memorizes every training example, but this solution will perform no better than chance on new data. The model has more than enough degrees of freedom to interpolate through every training point exactly, which is precisely what an unconstrained optimizer will eventually find if you train long enough. The key insight behind weight decay is that memorizing solutions tend to use large weights: the model must respond sharply to specific input patterns to correctly classify individual training examples, and sharp responses require large weight magnitudes.

Weight decay adds a regularization term directly to the loss:

where:

• $L(\theta)$: the original task loss (cross-entropy, MSE, etc.)
• $\lambda$: the weight decay coefficient, a positive hyperparameter controlling regularization strength
• $\|\theta\|_2^2$: the squared L2 norm of all parameters, defined as $\sum_i \theta_i^2$
• The factor $\frac{1}{2}$ is a mathematical convenience that cancels during differentiation

The gradient of this combined loss with respect to any parameter $\theta_i$ is:

where:

• $\frac{\partial L}{\partial \theta_i}$: the gradient of the task loss with respect to parameter $i$
• $\lambda \theta_i$: the regularization gradient, which is simply the parameter value scaled by $\lambda$

When you apply gradient descent with learning rate $\eta$, the update rule becomes:

Expanding this:

This formulation is revealing. The $(1 - \eta \lambda)$ factor multiplies the parameter before subtracting the gradient. Since $\eta \lambda$ is a small positive number (typically something like $0.01 \times 0.0001 = 10^{-6}$), you are shrinking the parameter slightly toward zero at every step before moving in the gradient direction. This is why weight decay is sometimes called L2 regularization and why it produces the "weight shrinkage" effect that gives the technique its name.

The shrinkage effect is cumulative. After $T$ steps with no gradient contribution, a weight $\theta$ would decay to:

For typical values of $\eta = 10^{-3}$ and $\lambda = 10^{-2}$, we have $\eta \lambda = 10^{-5}$, so after 10,000 steps without any gradient signal, a weight would retain $(1 - 10^{-5})^{10000} \approx e^{-0.1} \approx 0.905$ of its original magnitude. After 100,000 steps, it would retain about $e^{-1} \approx 0.37$. This slow but relentless decay is what keeps unused features from persisting indefinitely. Any weight that the task gradient does not consistently reinforce will eventually shrink toward zero.

The intuition behind penalizing large weights connects to the concept of function smoothness. A function implemented by a neural network with large weights responds strongly to small changes in input features. A small perturbation in the input can produce a large change in the output. This extreme sensitivity to inputs is the hallmark of a memorized solution: the model has learned to respond intensely to specific patterns in the training data, which may not generalize.

Consider a simple linear model $f(x) = w \cdot x + b$. If $w$ is very large, then a tiny change in $x$ produces a huge change in $f(x)$. The model is fragile. With L2 regularization, the optimizer must weigh the benefit of increasing $w$ to reduce training loss against the cost of increasing the $w^2$ penalty. This creates pressure toward smaller weights, which correspond to smoother, more robust functions.

There is also a probabilistic interpretation. Minimizing the L2-regularized loss is equivalent to maximum a posteriori (MAP) estimation with a Gaussian prior centered at zero on the weights. The prior expresses a belief that weights are likely to be small, and the regularization strength $\lambda$ controls how strongly you hold that prior belief relative to the evidence from the training data. Bayesian practitioners often think about weight decay in exactly these terms.

The geometry of this constraint is worth visualizing. In two-dimensional weight space, the L2 penalty defines concentric circular contours centered at the origin. The task loss defines its own contours centered at the unconstrained optimum. The regularized solution is found where the total loss surface, combining both, has its minimum. This generally pulls the solution toward the origin compared to the unregularized minimum.

This geometric view shows something important: the regularized solution is almost never exactly at the origin. The task gradient must be exactly zero for weight decay alone to drive a weight to zero, which happens only for completely useless parameters. For all parameters that contribute something to the task, the optimal regularized weight is a compromise between what the task demands and what the penalty allows. This is the key difference between L2 regularization and constraints: L2 regularization does not enforce a hard constraint on weight magnitude, it merely penalizes large magnitudes. The model retains the freedom to grow weights when the task gradient justifies the cost.

The connection between weight decay and Bayesian inference is more than a curiosity. It provides a rigorous justification for why weight decay is appropriate as a default regularizer and gives you a principled way to think about the strength of the prior.

In the Bayesian framing, you place a prior distribution $p(\theta)$ over model parameters before observing any data. A Gaussian prior centered at zero, $p(\theta_i) \propto \exp(-\theta_i^2 / (2\sigma^2))$, expresses the belief that weights are probably small and symmetric around zero. The maximum a posteriori (MAP) estimate is then:

Substituting the Gaussian prior:

Negating to turn this into a minimization:

The first term is the negative log-likelihood (the task loss), and the second term is exactly the L2 regularization penalty with $\lambda = 1/\sigma^2$. A small $\sigma^2$ (tight prior) corresponds to large $\lambda$ (strong regularization). A large $\sigma^2$ (diffuse prior) corresponds to small $\lambda$ (weak regularization). When $\sigma^2 \to \infty$, the prior becomes uninformative and weight decay disappears.

This probabilistic interpretation suggests a principled way to think about choosing $\lambda$: it reflects your prior belief about how large model weights should be. If you believe the task is well-suited to the model architecture and the data is representative, you might use a diffuse prior (small $\lambda$). If you are working with a model that is much larger than the training data warrants, a tighter prior (larger $\lambda$) is appropriate. This matches the empirical conventions described earlier: larger models relative to dataset size call for larger $\lambda$.

Weight decay uses the squared L2 norm, but a natural question is: why not the L1 norm instead? L1 regularization adds $\lambda \sum_i |\theta_i|$ to the loss, which has a fundamentally different effect on the learned parameters.

The gradient of the L1 penalty with respect to $\theta_i$ is $\lambda \cdot \text{sign}(\theta_i)$, a constant magnitude regardless of how large or small $\theta_i$ is. This means L1 regularization applies the same push toward zero whether a weight is 0.001 or 1000. The consequence is that L1 tends to produce exactly-zero weights for parameters that do not contribute to reducing the task loss, leading to sparse solutions.

L2 regularization, by contrast, has a gradient of $\lambda \theta_i$ that is proportional to the parameter's current value. Large weights receive strong push-back; small weights receive little. This gradient decreases as weights approach zero, which is why L2 shrinks weights toward zero without setting them exactly to zero. The result is dense solutions where most weights remain nonzero but are collectively kept small.

For neural network training, L2 (weight decay) is almost always preferred over L1. There are three main reasons. First, L2 is smooth and differentiable everywhere, making gradient-based optimization straightforward. L1 has a non-differentiable kink at zero, requiring subgradient methods or proximal operators. Second, dense solutions with uniformly small weights generalize well for neural networks, where many parameters contribute small amounts to the final prediction. Third, L1's sparsity-inducing property is less useful in neural networks than in linear models, since neural networks already have architectural mechanisms (neurons, attention heads) that provide a more interpretable form of functional sparsity.

There are settings where L1 is useful: when you want to prune weights explicitly, when working with linear models for feature selection, or when you want to induce embedding sparsity. But for end-to-end deep learning, L2 weight decay is the standard choice.

There is also a probabilistic analogy here. While L2 regularization corresponds to a Gaussian prior (smooth, symmetric), L1 regularization corresponds to a Laplace prior (heavier tails, sharper peak at zero). The Laplace prior places more probability mass exactly at zero, which is why L1 tends to produce exact zeros. The Gaussian prior places more probability mass at small but nonzero values, which is why L2 produces small but nonzero weights. For most neural network applications, the Gaussian prior is a better model of what we believe about the weight distribution.

To make the math concrete, let us trace through a single weight decay update for a scalar weight.

Suppose we have a single weight $\theta = 2.5$, a task loss gradient at this point of $\frac{\partial L}{\partial \theta} = -0.8$, a learning rate $\eta = 0.01$, and a weight decay coefficient $\lambda = 0.1$.

Without weight decay, the standard SGD update is:

The weight increases from 2.5 to 2.508 because the gradient is negative (the loss decreases as the weight increases).

With weight decay, the update adds the regularization gradient $\lambda \theta = 0.1 \times 2.5 = 0.25$:

The weight still increases, but by less. The task loss gradient wanted to push the weight up by 0.008, but the regularization gradient is opposing this by adding a positive component (0.0025) to the total gradient update in the direction of shrinking the weight.

Equivalently, using the shrinkage form:

The two forms give the same result, as expected. The shrinkage form makes visible that the weight is first multiplied by 0.999 (a 0.1% reduction) and then adjusted by the task gradient. Over 1000 steps with no gradient contribution, the weight would decay from 2.5 toward zero following a geometric series: $2.5 \times 0.999^{1000} \approx 2.5 \times 0.368 \approx 0.92$.

This exponential decay toward zero is where the name "weight decay" originates. At every step, the optimizer applies a small multiplicative shrinkage to each weight, continuously counteracting the tendency for weights to grow large. The task gradient can push weights away from zero when a feature improves the predictions, but it must constantly overcome the decay pressure to do so.

There is an instructive way to think about what weight decay does to the steady-state weight magnitude. Suppose a weight has converged and is no longer changing on average. At that point, the average contribution of the task gradient must exactly cancel the contribution of the weight decay term:

This means the steady-state magnitude of the weight is determined by the ratio of the expected task gradient to the weight decay coefficient:

A stronger weight decay (larger $\lambda$) forces weights to be smaller in equilibrium, which is only possible if the gradient signal for that weight is also stronger. Parameters that contribute little to reducing the task loss (small gradients) will be pushed toward zero. Parameters that contribute strongly (large gradients for important features) will stabilize at larger values. Weight decay thus implements a soft version of importance weighting: only weights that are useful enough to overcome the decay pressure remain large.

This equilibrium perspective also clarifies why weight decay cannot cause catastrophic collapse. If the weight decay were too aggressive and drove all weights toward zero, the model would make random predictions, generating large gradients that push weights back up. The equilibrium is stable: the task gradient always provides a restoring force proportional to how poorly the model performs. Weight decay shifts the equilibrium point but cannot collapse the model to zero as long as the task has a learnable signal.

To see why large weights are the mechanism behind overfitting (rather than just a symptom), consider what happens during unconstrained gradient descent on a small dataset. The optimizer follows a path of steepest descent in the loss landscape, always moving in the direction that most rapidly reduces training loss. Early in training, this is fine: both training and validation loss decrease as the model learns recurring patterns. But as the model approaches the training data manifold, the loss surface develops sharp minima that correspond to memorizing individual training examples.

These sharp minima are characterized by large weights. The model learns to respond with high confidence to specific training examples by amplifying the weights connecting the relevant input features to the output. This amplification is invisible to the optimizer, which sees only that training loss continues to decrease. But you can see the consequence in the weight norms: they grow throughout training as the model memorizes more and more.

Weight decay changes the shape of the loss surface. The penalty term adds a bowl-shaped component to the total loss, converting sharp minima into smoother, broader minima. The optimizer can no longer rest in narrow memorization minima because the cost of staying there (the penalty for large weights) exceeds the benefit (the reduction in training loss). Instead, it settles in broader minima that correspond to learned patterns rather than memorized examples. These broader minima are precisely the solutions that generalize well to unseen data.

Standard weight decay works beautifully with stochastic gradient descent. The math lines up: adding the L2 penalty to the loss is exactly equivalent to decaying the weights by a fixed fraction at each step. However, when you move to adaptive gradient methods like Adam, this equivalence breaks down in a way that significantly matters in practice.

Before examining the problem, it helps to recall how Adam updates parameters. Adam maintains per-parameter adaptive learning rates. For each parameter $\theta_i$, it tracks the first moment (exponential moving average of gradients) $m_i$ and the second moment (exponential moving average of squared gradients) $v_i$. The full update procedure at step $t$ is:

where:

• $g_i$: the gradient of the loss with respect to parameter $i$
• $\beta_1$: decay rate for the first moment (typically 0.9)
• $\beta_2$: decay rate for the second moment (typically 0.999)
• $\hat{m}_i$: bias-corrected first moment estimate, correcting for initialization at zero
• $\hat{v}_i$: bias-corrected second moment estimate
• $\epsilon$: small constant for numerical stability (typically $10^{-8}$)

The key insight about Adam is that the denominator $\sqrt{\hat{v}_i} + \epsilon$ normalizes the update by the root mean square of past gradients. Parameters that have historically large gradients receive smaller updates (the large denominator divides them down), while parameters with historically small gradients receive larger updates. This per-parameter rescaling is what makes Adam so effective for neural networks, where different parameters may have very different natural gradient scales.

When you add L2 regularization to the loss, the gradient passed to Adam becomes $g_i + \lambda \theta_i$, where $g_i$ is the task loss gradient. This regularization gradient gets folded into both the first and second moment estimates:

As a result, the adaptive learning rate scale, which is computed from gradient history, is influenced by the regularization term. The second moment $v_i$ now reflects the history of both the task gradient and the weight decay gradient.

The consequence is that the effective weight decay depends on the history of gradients for that parameter. Parameters with small gradients from the task loss will have small $v_i$ values. Adam would normally amplify their updates (small denominator). But because $\lambda \theta_i$ is also folded into $v_i$, the second moment estimate is inflated by the regularization term, partially suppressing this amplification. The actual regularization strength varies in complicated ways across parameters, defeating the purpose of having a clean $\lambda$ hyperparameter.

The deeper problem is this: the adaptive scaling that Adam applies to regularize the step size based on gradient history should apply to the task signal, not to the weight decay signal. Weight decay is supposed to be a global shrinkage applied uniformly to all parameters. When it gets folded into Adam's adaptive mechanism, the global uniformity is lost. Parameters with small task gradients receive weaker regularization, which is the opposite of what you want: these are often the parameters most likely to drift to unusual values, since Adam amplifies their updates.

Ilya Loshchilov and Frank Hutter identified this problem in their 2017 paper "Decoupled Weight Decay Regularization" and proposed a fix: separate the weight decay step from the gradient update step.

AdamW modifies the Adam update by applying weight decay directly to the parameters, independently of the adaptive gradient scaling. The update rule becomes:

where:

• $g_i$: the gradient of the task loss (not including the regularization term)
• $\lambda \theta_i$: the weight decay term applied directly to the parameter, outside the adaptive scaling
• All other terms are as in standard Adam

The critical difference is in the last line. The weight decay term $\lambda \theta_i$ is added to the update outside the adaptive denominator $\sqrt{\hat{v}_i} + \epsilon$. This means the weight decay applies uniformly to all parameters at the same rate $\lambda$, regardless of their gradient history. The adaptive learning rate only affects the gradient term, not the regularization term.

Expanding the AdamW update:

This is analogous to the SGD case: multiply the parameter by $(1 - \eta \lambda)$ to shrink it, then subtract the adaptively scaled gradient. The weight decay effect is clean and uniform.

In practice, AdamW consistently outperforms Adam with L2 regularization for training large language models and other deep networks. The reason is that uniform weight decay across all parameters is a more coherent regularization strategy. When Adam applies L2 regularization internally, parameters with small gradients (often those that represent rarely-used features or less-frequently updated neurons) receive weaker regularization. These are exactly the parameters most at risk of drifting to unusual values, since the optimizer's adaptive scaling amplifies their updates. AdamW avoids this asymmetry.

The practical difference is especially visible when you tune the weight decay hyperparameter. With standard Adam + L2, the optimal $\lambda$ tends to be entangled with the learning rate in a complex way, making hyperparameter search difficult. AdamW decouples these, making $\lambda$ and $\eta$ more independently tunable and simplifying the optimization landscape.

A concrete way to see the difference: suppose you have a parameter $\theta_i$ with consistently small task gradients, so $v_i$ is small (Adam will amplify its updates). With L2 Adam, the regularization term $\lambda \theta_i$ gets divided by the small $\sqrt{v_i}$, making the effective regularization on this parameter much larger than $\lambda$ would suggest. With AdamW, the regularization is exactly $\eta \lambda \theta_i$ regardless of $v_i$. This predictability is what makes AdamW a reliable optimizer for large-scale training.

To put numbers on this: if $\sqrt{v_i} = 0.001$ (a rarely-updated parameter), $\eta = 0.0001$, $\lambda = 0.01$, and $\theta_i = 1.0$, then:

• With L2 Adam: effective weight decay per step is approximately $\eta \lambda \theta_i / \sqrt{v_i} = 0.0001 \times 0.01 \times 1.0 / 0.001 = 0.001$ (100 times stronger than intended)
• With AdamW: effective weight decay per step is $\eta \lambda \theta_i = 0.0001 \times 0.01 \times 1.0 = 0.000001$ (exactly as intended)

The L2 Adam approach accidentally applies extremely strong regularization to rarely-updated parameters, which is exactly backwards from what a well-calibrated regularizer should do. AdamW keeps regularization proportional to $\lambda$ throughout, regardless of the parameter's gradient history.

It might seem surprising that a flaw in how L2 regularization interacts with Adam went unnoticed for several years after Adam was introduced in 2014. The reason is that for many tasks, the difference between L2 Adam and AdamW is modest: both apply some form of regularization, and both work better than no regularization at all. The difference becomes most visible when you are carefully tuning hyperparameters for large-scale pretraining, where the precise meaning of the weight decay coefficient matters and the scale of training amplifies small differences. The natural language processing community discovered the practical importance of AdamW through large-scale BERT and GPT training, where the cleaner regularization semantics of AdamW translated into measurable improvements.

The Loshchilov and Hutter paper also documented a related issue: the optimal learning rate for Adam with L2 regularization depends on the weight decay coefficient, because both affect the effective step size. This coupling makes grid search over hyperparameters much harder. AdamW breaks this coupling, making hyperparameter search over $\eta$ and $\lambda$ more independent.

An important practical question is whether to apply weight decay uniformly to all parameters or selectively. In most modern training setups, weight decay is not applied to all parameters equally.

Certain parameters are typically excluded from weight decay:

Bias terms. Bias parameters shift the activation values of neurons. They do not contribute to the learned feature magnitudes in the same way as weight matrices. Applying weight decay to biases can interfere with the model's ability to represent the correct output scale, and in practice it rarely helps. The bias is a low-dimensional additive offset, and regularizing it toward zero can systematically miscalibrate outputs.

Layer normalization parameters. Layer normalization layers have learned scale ($\gamma$) and shift ($\beta$) parameters. These control the distribution of activations after normalization. Applying weight decay to them can destabilize training by preventing the normalization from correctly calibrating the output scale. In practice, this is one of the more commonly overlooked mistakes that causes subtle training instability.

Embeddings (sometimes). Embedding parameters are weight matrices that map discrete tokens or positions to continuous vectors. Some practitioners exclude them from weight decay because they can be viewed as lookup tables rather than function weights, though this is less universally agreed upon. For large pretraining runs, it is common to include embeddings in weight decay, but for fine-tuning, excluding them or using a very small $\lambda$ for embeddings can prevent degrading carefully trained representations.

Excluding bias and normalization parameters measurably improves training stability and final performance. Most published training configurations for transformers explicitly separate parameters into "weight decay" and "no weight decay" groups before passing them to the optimizer.

A simple and widely used heuristic separates parameters by their dimensionality: parameters with more than one dimension receive weight decay, while one-dimensional parameters do not. Weight matrices are two-dimensional and should receive decay. Bias vectors are one-dimensional and should not. Layer normalization scale and shift vectors are one-dimensional and should not. Embedding matrices are two-dimensional and typically do.

The logic behind this heuristic is that one-dimensional parameters are almost always either offsets (biases) or normalization scalings, which have a different role in the model than the weight matrices that transform feature representations.

The output shows exactly which parameters are excluded. Biases and LayerNorm parameters, which have one dimension, are excluded from weight decay. Weight matrices, which are two-dimensional, receive weight decay.

To understand why excluding biases matters, consider what weight decay does to a bias term. Suppose a layer has bias $b = 3.0$ that helps shift the activations to the correct scale for the downstream computation. With weight decay, at each step the bias is multiplied by $(1 - \eta \lambda)$, shrinking it toward zero. Over training, this constant pressure toward zero systematically biases the model's activations in the wrong direction. The task loss gradient will partially compensate, but the result is a tug-of-war where the bias never fully settles at its optimal value. The training dynamic becomes less stable and the final model may have systematically miscalibrated outputs. This effect is subtle in small models but can accumulate significantly in deep networks with many layers.

The same argument applies even more forcefully to layer normalization parameters. The $\gamma$ and $\beta$ parameters of LayerNorm are initialized to values that keep the normalized distribution at unit scale and zero mean. If weight decay shrinks $\gamma$ toward zero, it counteracts the normalization, reducing the effective signal magnitude layer by layer. In deep transformers, this can produce training instability that resembles a learning rate problem but is caused by misapplied regularization.

Selecting $\lambda$ is a hyperparameter tuning problem, but there are well-established conventions and principles that narrow the search space considerably.

The weight decay coefficient $\lambda$ is typically set in the range $10^{-4}$ to $10^{-1}$, with $10^{-2}$ being a common default for many transformer training runs.

The BERT paper used weight decay of $0.01$. GPT-3 used $0.1$. Many recent models use values in the $0.01$ to $0.1$ range. The right value depends on the dataset size, model size, and how much regularization is needed:

• Small datasets, large models: Higher weight decay (closer to $0.1$) because the model has more capacity to overfit.
• Large datasets, appropriately sized models: Lower weight decay (closer to $0.01$) because the data itself provides strong regularization.
• Very large pretraining runs: Weight decay is often moderate ($0.01$ to $0.1$) since the sheer volume of data already prevents significant overfitting, but the penalty helps maintain parameter regularity.

Setting $\lambda = 0.0$ (no weight decay) is a valid and sometimes correct choice. If you are fine-tuning a large pretrained model on a small downstream task, the pretrained weights already encode a strong prior about what good representations look like. Aggressive weight decay can hurt by pulling carefully learned features back toward zero. In such fine-tuning scenarios, a modest $\lambda$ around $10^{-3}$ or no weight decay at all is often appropriate. Treat zero weight decay as a baseline to test against, not an oversight.

When using AdamW, the effective weight decay per step is $\eta \lambda$, where $\eta$ is the learning rate. This means weight decay and learning rate are coupled in their effect: if you increase the learning rate by $10\times$, the weight decay effect per step also increases by $10\times$.

This coupling is why some researchers prefer to think about normalized weight decay defined as $\lambda / \sqrt{\text{steps}}$ or to schedule weight decay alongside the learning rate. If your learning rate follows a cosine decay schedule, and weight decay is held constant, the effective regularization per step decreases as the learning rate decreases. Some training setups adjust weight decay to compensate for this, though it is not universally done.

A useful way to think about this: during the warmup and peak learning rate phase of training, your model experiences the strongest regularization per step. As the learning rate decays, regularization weakens. This provides a reasonable inductive bias: early in training, strong regularization prevents the model from memorizing noise in the early data it sees. Later in training, as the model converges, lighter regularization allows it to fine-tune toward the optimal weights without constant shrinkage pressure fighting against it.

In practice, the simplest approach is to tune $\lambda$ jointly with the learning rate during a hyperparameter search, or to use values from published configurations as starting points and adjust from there.

Fine-tuning deserves special attention because the regularization goals are different from pretraining. When you pretrain a language model from scratch, you want weight decay to prevent memorization and encourage general representations. When you fine-tune a pretrained model on a specific task, you want to preserve the pretrained representations while adapting them to the new task.

This creates a tension. Aggressive weight decay during fine-tuning pulls the weights back toward zero, not toward the pretrained values. The pretrained weights are not at zero; they encode rich, learned features at various magnitudes. Decay toward zero can erase valuable pretrained representations faster than the fine-tuning gradient can reinstate them.

There are two approaches to handling this. The first is to use a very small or zero weight decay during fine-tuning. The second, more principled approach is L2-SP (L2 starting point regularization), which penalizes deviation from the pretrained weights rather than from zero:

This preserves the pretrained representations while still preventing large unconstrained updates. We covered catastrophic forgetting in the finetuning fundamentals chapters; L2-SP is one technique that directly addresses it. Standard weight decay does not distinguish between "moving away from zero" and "moving away from the pretrained values," which is why it can hurt fine-tuning when applied naively.

In practice, most practitioners fine-tune with a very small weight decay (around $10^{-4}$ to $10^{-3}$) rather than zero or full L2-SP, accepting a small regularization bias toward zero as a tradeoff for implementation simplicity. The choice matters more for small downstream datasets (a few hundred to a few thousand examples) where the fine-tuning gradient is weak. For larger downstream datasets (tens of thousands of examples or more), the fine-tuning gradient is strong enough to override the small decay pressure, and the distinction between standard weight decay and L2-SP becomes less important.

This section walks through a complete implementation demonstrating weight decay in practice: comparing the effect of different $\lambda$ values on a small neural network and showing how AdamW differs from Adam with L2 regularization.

We will train a small multi-layer perceptron on a regression task with limited data, where overfitting is likely without regularization.

The training set is intentionally small (80 samples) to make overfitting visible. The true function combines sine and cosine components, which a neural network can approximate, but with only 80 noisy observations there is ample room to overfit. We add Gaussian noise with standard deviation 0.3 to simulate the measurement uncertainty common in real-world data. A four-layer MLP with 128 hidden units has far more parameters than the 80 training points, making it the ideal candidate to demonstrate overfitting.

One more visualization helps build intuition about how weight decay affects individual weights over the course of training. We can track a single weight's value over training epochs to see the interplay between gradient updates and the decay pressure.

The trajectory plot shows how decay pressure interacts with, rather than simply overrides, the task gradient. The independently trained models follow different paths, so their norms are not monotonically ordered by $\lambda$ at every epoch. The strongest setting, $\lambda=0.1$, is the only curve that turns downward substantially after reaching its peak, while the lower-decay runs continue growing or level off.

Notice that none of the equilibrium norms is zero: each reflects a balance between the expected task gradient and the weight decay term. The flattening portions show the models approaching that balance, while continued growth or decline shows that the two forces have not yet settled.

The practical application of weight decay in transformer-based language models involves several details that go beyond the basic recipe. Published training recipes show both common parameter choices and the reasoning behind them.

Examining how major language models have used weight decay provides useful reference points.

BERT (2018) used $\lambda = 0.01$ with AdamW for pretraining, with bias and LayerNorm parameters excluded. This configuration became one of the most influential templates for transformer pretraining. The relatively modest $\lambda = 0.01$ reflects the large training corpus (BookCorpus plus Wikipedia, roughly 3 billion tokens), where the data itself provides strong regularization against memorization.

GPT-3 (2020) used $\lambda = 0.1$ with AdamW. This stronger regularization was chosen despite the massive training corpus (300 billion tokens) because GPT-3 has 175 billion parameters, far exceeding the number of training tokens. The high model-to-data capacity ratio justified stronger weight decay.

LLaMA (2023) used $\lambda = 0.1$ with AdamW. LLaMA was trained on roughly 1 trillion tokens with models ranging from 7B to 65B parameters, a much more compute-efficient regime. The $\lambda = 0.1$ choice suggests that even with large data quantities, the training team preferred stronger regularization for large models.

PaLM (2022) used $\lambda = 0.1$ with Adafactor, a memory-efficient adaptive optimizer. This reinforces the trend toward $\lambda = 0.1$ for very large models, regardless of the specific optimizer variant.

The pattern across these configurations is clear: as model size grows relative to dataset size, stronger weight decay is preferred. The $0.01$ to $0.1$ range covers most practical scenarios, with $0.1$ being increasingly common for billion-parameter models.

A practical tip when starting a new project: rather than searching over a wide range, start with $\lambda = 0.01$ as your default. If you observe clear signs of overfitting (train loss much lower than validation loss), increase to $0.05$ or $0.1$. If you observe underfitting (both train and validation loss are high and adding more epochs does not help training loss), reduce to $0.001$ or remove weight decay entirely. This three-point search covers the most important range without expensive grid searches.

Weight decay and gradient clipping are both tools for keeping the optimization process well-behaved, and they interact in subtle ways. Gradient clipping (covered in Chapter 13) caps the gradient norm to prevent explosive updates. Weight decay prevents weights from growing large over time.

In practice, most transformer training recipes use both together. Gradient clipping handles sudden large gradient spikes that can destabilize training. Weight decay handles the gradual tendency for weights to grow large over many steps of stable training. Neither fully substitutes for the other.

When gradient clipping is active and clips a gradient, it scales down all gradient components proportionally. This means the task gradient term in the AdamW update is clipped, but the weight decay term $\eta \lambda \theta_i$ is not affected by clipping. This behavior is desirable: weight decay continues to apply its steady shrinkage pressure even during gradient spikes, preventing the weight magnitudes from growing large precisely when gradient instability might push them in that direction.

A natural question is: if gradient clipping prevents explosive gradient steps, and weight decay prevents large weights, do you really need both? The answer is yes, because they address different timescales. Gradient clipping is a reactive mechanism that fires when a gradient is too large at a specific step. Weight decay is a proactive mechanism that continuously prevents the slow accumulation of large weights over thousands of steps. A model trained with only gradient clipping can still develop large weights gradually; each individual step is clipped, but the cumulative effect of many moderate steps can still push weights to large magnitudes. Weight decay prevents this slow drift.

Weight decay is not the only regularization technique available, and understanding where it sits relative to alternatives helps you choose the right tool for different situations.

Dropout randomly zeros activations during the forward pass, which prevents neurons from co-adapting. It is most effective for fully-connected layers and is less useful for convolutional layers (where spatial correlations are expected) or attention layers (where the patterns should be consistent). Weight decay, by contrast, applies everywhere in the network.

Data augmentation increases the effective dataset size by transforming training examples. It addresses overfitting at the data level rather than the model level. Combining weight decay with data augmentation is common and effective: data augmentation increases training diversity, while weight decay constrains the model's capacity to memorize.

Batch normalization was once used as an implicit regularizer, since its stochastic nature (batch statistics vary per mini-batch) provides a form of regularization. However, modern transformers rely on layer normalization, which has less regularization effect. For these architectures, explicit weight decay is more important.

Early stopping monitors validation loss and stops training before overfitting worsens. This is compatible with weight decay but is a less principled approach: early stopping discards training steps that might still be beneficial, while weight decay allows full training by constraining the solution space.

For transformer-based language models, the typical recipe combines weight decay with dropout (especially in the attention and feed-forward layers), data preprocessing and augmentation at the corpus level, and no early stopping (since training to convergence is generally better when weight decay is calibrated correctly). This combination works reliably across many model scales and tasks.

A practical complication arises when combining weight decay with learning rate schedules. As discussed in earlier chapters, most large-scale training runs use a warmup period followed by cosine or linear decay of the learning rate. With AdamW, the effective weight decay per step is $\eta \lambda$. If $\eta$ varies by a factor of 1000 over training (e.g., from a peak of $3 \times 10^{-4}$ to a minimum of $3 \times 10^{-7}$), the effective regularization per step also varies by a factor of 1000.

This means the model experiences very different amounts of regularization at different phases of training. During the high-learning-rate phase, the combined effect of large gradient steps and large weight decay steps can provide strong regularization but also high noise. During the low-learning-rate phase, both the gradient updates and the weight decay become very small, and the model is essentially making tiny adjustments with minimal regularization.

Some research groups have proposed learning rate-independent weight decay, where $\lambda$ is adjusted to keep the effective decay rate constant relative to the parameter update magnitude. In practice, this adds complexity without consistent benefit for most training runs. The standard approach remains using a fixed $\lambda$ throughout training and accepting that the effective per-step regularization varies with the learning rate schedule.

A related question is whether to use a weight decay schedule: starting high early in training and reducing it later. Intuitively, you might want strong regularization early (when the model is still far from any good solution and vulnerable to memorizing noise) and weaker regularization late (when you want precise fine-tuning). Some training setups experiment with this, but it adds another hyperparameter dimension and is rarely used in published large-scale training. The consensus is that a fixed $\lambda$ combined with a learning rate schedule already implicitly provides this behavior: since $\eta$ decreases over training, the effective per-step weight decay $\eta \lambda$ also decreases, achieving something similar to a decaying weight decay schedule without explicitly coding it.

Weight decay is a powerful regularizer, but it is not without limitations. Understanding when it helps and when it falls short is essential for effective use.

It does not help when data is the bottleneck. Weight decay reduces overfitting, but if your model is underfitting (high training and validation loss), adding weight decay will make things worse, not better. Before applying regularization, confirm that your model can achieve low training loss without it. If training loss is already high, the problem is model capacity or optimization, not generalization. A common mistake is to add weight decay when validation loss is high without first checking whether training loss is also high. If both are high, the model needs more capacity or better optimization, not more regularization.

Choosing the wrong parameters to regularize degrades performance. Applying weight decay to biases and normalization parameters (as discussed earlier) can subtly degrade both training stability and final performance. If you are using a generic training loop that applies weight decay to all parameters without filtering, you may be accidentally regularizing the wrong things. Always use parameter groups to separate the parameters that should and should not receive weight decay. This is a single function call in PyTorch and a few lines of code, but it is one of the most consistently overlooked best practices in training transformers from scratch.

Weight decay is not a substitute for architecture choices. Some overfitting problems stem not from lack of regularization but from poor architecture choices, such as using a model that is far too large for the task, or using a model that lacks appropriate inductive biases. Weight decay can mask architectural issues by providing additional regularization capacity, but the underlying mismatch between model capacity and task complexity remains. When you find that you need very large $\lambda$ to prevent overfitting, consider whether reducing model size, using a better pretraining objective, or collecting more data would be more appropriate solutions.

The interaction with learning rate scheduling requires attention. Because the effective weight decay per step is $\eta \lambda$, any change to the learning rate affects the regularization strength. If you use a cosine learning rate schedule that reduces $\eta$ by a factor of 100 over training, the effective weight decay per step also reduces by 100. In some settings, this is desirable (the model should be less constrained as it fine-tunes). In others, it creates an inconsistent regularization profile. Some practitioners use a constant effective weight decay by adjusting $\lambda$ proportionally as $\eta$ changes, though this adds complexity.

It penalizes all weight magnitudes equally. Weight decay does not distinguish between important and unimportant weights. A weight that is large because it captures an important predictive feature receives the same penalty as one that is large due to overfitting. Techniques like L1 regularization (which induces sparsity) or structured pruning offer more targeted alternatives, but they come with their own complications in optimization.

It interacts with batch normalization in complex ways. When batch normalization is present, the effective regularization from weight decay is complicated by the fact that batch norm rescales activations. A weight matrix of any magnitude produces the same normalized activations, so weight decay can push weights toward zero without directly affecting the function the model computes. This can lead to training instabilities and is one reason why some architectures have moved toward layer normalization, where this interaction is less problematic. Modern transformers use pre-norm with layer normalization rather than batch normalization, which largely sidesteps this issue.

Fine-tuning requires different treatment. Applying standard weight decay during fine-tuning pulls pretrained weights toward zero rather than toward their pretrained values. This can erase valuable pretrained representations, particularly when fine-tuning on small datasets where the task gradient is weak. In fine-tuning settings, either using no weight decay, a very small $\lambda$, or switching to L2-SP regularization (which penalizes deviation from pretrained weights) is often preferable.

It provides no sparsity. Unlike L1 regularization, weight decay never drives individual weights exactly to zero. This means weight decay does not help with model compression, pruning, or interpretability through sparse feature selection. If your goal is a smaller or more interpretable model rather than better generalization, weight decay alone is insufficient. You would need to combine it with explicit sparsity-inducing techniques.

Despite these limitations, weight decay remains one of the most universally applied regularization techniques in deep learning. It is easy to implement, computationally free (the penalty gradient requires negligible computation compared to the forward pass), and reliably improves generalization when tuned appropriately. Every major language model training pipeline uses weight decay with AdamW as a core component of the training recipe. The consistent appearance of $\lambda \in [0.01, 0.1]$ across BERT, GPT-3, LLaMA, PaLM, and dozens of other published models reflects a consensus that this simple technique delivers reliable improvements without requiring complex tuning.

Weight decay adds a penalty proportional to the squared magnitude of model weights to the training loss, discouraging large weights and encouraging solutions that generalize better. The key ideas from this chapter are:

• L2 regularization adds $\frac{\lambda}{2} \|\theta\|_2^2$ to the loss, producing a $(1 - \eta\lambda)$ shrinkage factor on weights at each update step. This is mathematically equivalent to weight decay for SGD but not for adaptive optimizers.
• Decoupled weight decay (AdamW) applies the weight decay term directly to parameters rather than through the loss gradient, making sure uniform regularization across all parameters regardless of their gradient history. This outperforms standard Adam with L2 regularization for most large-scale training setups.
• Selective application matters: bias terms and layer normalization parameters are typically excluded from weight decay to preserve training stability. The standard heuristic excludes all one-dimensional parameters.
• Coefficient selection typically starts around $\lambda = 0.01$ for transformer training and is tuned based on dataset size and model capacity, with larger values for smaller datasets. Published configurations from BERT ($0.01$), GPT-3 ($0.1$), and LLaMA ($0.1$) provide useful starting points.
• The learning rate coupling means effective regularization per step scales with both $\lambda$ and the learning rate, which is worth considering when using aggressive learning rate schedules.
• Fine-tuning requires care: standard weight decay pulls weights toward zero, not toward the pretrained initialization, which can hurt fine-tuning on small datasets. Use smaller $\lambda$ or L2-SP regularization when adapting pretrained models.

The next chapter covers gradient accumulation, another training optimization technique that addresses the practical constraint of GPU memory limits by simulating large batch training through accumulated gradient updates.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about weight decay and AdamW.