Weight Decay › Choosing the Weight Decay Coefficient

Selecting $\lambda$ is a hyperparameter tuning problem, but there are well-established conventions and principles that narrow the search space considerably.

The weight decay coefficient $\lambda$ is typically set in the range $10^{-4}$ to $10^{-1}$, with $10^{-2}$ being a common default for many transformer training runs.

The BERT paper used weight decay of $0.01$. GPT-3 used $0.1$. Many recent models use values in the $0.01$ to $0.1$ range. The right value depends on the dataset size, model size, and how much regularization is needed:

• Small datasets, large models: Higher weight decay (closer to $0.1$) because the model has more capacity to overfit.
• Large datasets, appropriately sized models: Lower weight decay (closer to $0.01$) because the data itself provides strong regularization.
• Very large pretraining runs: Weight decay is often moderate ($0.01$ to $0.1$) since the sheer volume of data already prevents significant overfitting, but the penalty helps maintain parameter regularity.

Setting $\lambda = 0.0$ (no weight decay) is a valid and sometimes correct choice. If you are fine-tuning a large pretrained model on a small downstream task, the pretrained weights already encode a strong prior about what good representations look like. Aggressive weight decay can hurt by pulling carefully learned features back toward zero. In such fine-tuning scenarios, a modest $\lambda$ around $10^{-3}$ or no weight decay at all is often appropriate. Treat zero weight decay as a baseline to test against, not an oversight.

When using AdamW, the effective weight decay per step is $\eta \lambda$, where $\eta$ is the learning rate. This means weight decay and learning rate are coupled in their effect: if you increase the learning rate by $10\times$, the weight decay effect per step also increases by $10\times$.

This coupling is why some researchers prefer to think about normalized weight decay defined as $\lambda / \sqrt{\text{steps}}$ or to schedule weight decay alongside the learning rate. If your learning rate follows a cosine decay schedule, and weight decay is held constant, the effective regularization per step decreases as the learning rate decreases. Some training setups adjust weight decay to compensate for this, though it is not universally done.

A useful way to think about this: during the warmup and peak learning rate phase of training, your model experiences the strongest regularization per step. As the learning rate decays, regularization weakens. This provides a reasonable inductive bias: early in training, strong regularization prevents the model from memorizing noise in the early data it sees. Later in training, as the model converges, lighter regularization allows it to fine-tune toward the optimal weights without constant shrinkage pressure fighting against it.

In practice, the simplest approach is to tune $\lambda$ jointly with the learning rate during a hyperparameter search, or to use values from published configurations as starting points and adjust from there.

Fine-tuning deserves special attention because the regularization goals are different from pretraining. When you pretrain a language model from scratch, you want weight decay to prevent memorization and encourage general representations. When you fine-tune a pretrained model on a specific task, you want to preserve the pretrained representations while adapting them to the new task.

This creates a tension. Aggressive weight decay during fine-tuning pulls the weights back toward zero, not toward the pretrained values. The pretrained weights are not at zero; they encode rich, learned features at various magnitudes. Decay toward zero can erase valuable pretrained representations faster than the fine-tuning gradient can reinstate them.

There are two approaches to handling this. The first is to use a very small or zero weight decay during fine-tuning. The second, more principled approach is L2-SP (L2 starting point regularization), which penalizes deviation from the pretrained weights rather than from zero:

$$L_{\text{total}}(\theta) = L(\theta) + \frac{\lambda}{2} \|\theta - \theta_{\text{pretrained}}\|_2^2$$

This preserves the pretrained representations while still preventing large unconstrained updates. We covered catastrophic forgetting in the finetuning fundamentals chapters; L2-SP is one technique that directly addresses it. Standard weight decay does not distinguish between "moving away from zero" and "moving away from the pretrained values," which is why it can hurt fine-tuning when applied naively.

In practice, most practitioners fine-tune with a very small weight decay (around $10^{-4}$ to $10^{-3}$) rather than zero or full L2-SP, accepting a small regularization bias toward zero as a tradeoff for implementation simplicity. The choice matters more for small downstream datasets (a few hundred to a few thousand examples) where the fine-tuning gradient is weak. For larger downstream datasets (tens of thousands of examples or more), the fine-tuning gradient is strong enough to override the small decay pressure, and the distinction between standard weight decay and L2-SP becomes less important.