Weight Decay › Weight Decay

Neural networks have an enormous number of parameters. A modern large language model can have hundreds of billions of them, but even a modest classifier trained on a small dataset might have millions. With that many degrees of freedom, the network can memorize training data rather than learn to generalize. It fits noise, quirks of the specific examples it saw, and spurious patterns that do not hold in the real world. This failure mode is called overfitting, and combating it is one of the central challenges in training.

Weight decay is one of the oldest and most effective tools for preventing overfitting. The idea is simple: penalize large weights. A model with large weights is a model that depends heavily on specific input features, which is often a sign of memorization rather than generalization. By adding a penalty term to the loss function that grows with the magnitude of the weights, you encourage the optimizer to prefer simpler, smoother solutions that are more likely to transfer to unseen data.

Weight decay sits alongside other regularization techniques like dropout, which was covered in Part X. While dropout works by randomly zeroing activations during training, weight decay operates directly on the loss function, pushing parameter values toward zero throughout training. The two techniques are complementary and are often used together. Their combination addresses two different sources of overfitting: dropout prevents co-adaptation of neurons, while weight decay prevents any individual weight from growing large enough to dominate the model's behavior.

As we covered in the chapters on learning rate warmup, learning rate decay, cosine schedules, and large batch training, training optimization involves many interacting decisions. Weight decay is another lever in this system, one that is particularly important when working with large models on limited data, or when using optimizers like Adam where the interaction between weight decay and adaptive gradient scaling requires special care. Getting weight decay right can mean the difference between a model that generalizes cleanly and one that memorizes training quirks.

This chapter develops weight decay from first principles: why large weights are problematic, how the L2 penalty addresses this at both the loss-function level and the parameter update level, why standard L2 regularization fails to work as intended with adaptive optimizers like Adam, how AdamW fixes this with decoupled weight decay, and how to apply these ideas correctly in practice. By the end, you will understand how to set a weight decay coefficient, why you are setting it, and what it does to the model's learning dynamics.