AdamW › AdamW: Decoupled Weight Decay for Neural Networks

Adam became the default optimizer for deep learning almost immediately after its 2014 publication. It combines momentum with adaptive learning rates, and for most tasks, it simply works. Practitioners adopted it widely, and it quickly became the go-to choice for training language models, image classifiers, and a range of other architectures. Yet there is a subtle flaw in how Adam handles regularization, one that went largely unnoticed for three years until Ilya Loshchilov and Frank Hutter pointed it out in their 2017 paper "Decoupled Weight Decay Regularization." The fix they proposed, AdamW, is now the standard optimizer for training large language models.

The flaw is not an edge case or a minor numerical issue. It is a fundamental mathematical inconsistency between what practitioners intend when they set a "weight decay" parameter and what Adam actually computes. Understanding this inconsistency requires stepping back from the optimizer itself and examining what regularization is trying to do in the first place, and then tracing exactly how Adam distorts that intent.

This chapter walks through that complete arc, from the motivation for regularization in neural networks to the precise mathematical analysis of why Adam handles it incorrectly, through the AdamW formulation and its practical implications for LLM training. By the end, you will have a solid understanding of how to use AdamW, why it was necessary, and what it achieves.