AdamW › Limitations and Impact

AdamW is a meaningful improvement over Adam with L2 regularization, but it inherits many of Adam's properties and does not solve all optimization challenges. Understanding where AdamW helps, where it does not, and why it became the standard for language model training gives you a complete picture of its role in the deep learning toolkit.

AdamW corrects the mathematical inconsistency of L2 regularization inside Adam, making the weight decay hyperparameter behave as practitioners intend. This is the fundamental contribution of the Loshchilov-Hutter paper: not a dramatic algorithmic innovation, but a precise identification of an existing inconsistency and a minimal fix that restores the intended behavior.

What AdamW does not fix is the general challenge of tuning regularization in deep learning. The optimal weight decay coefficient still requires tuning, and its interaction with the learning rate schedule adds complexity. A cosine decay schedule effectively reduces weight decay's influence in later training stages as the learning rate decreases, which can be desirable (allowing the model to converge cleanly at the end) or undesirable depending on the training regime. There is no automatic way to set weight decay; it remains an important hyperparameter that requires experimental validation.

AdamW also does not resolve some of Adam's other known limitations. There exist convergence results showing that Adam can converge to suboptimal solutions in certain settings, particularly on non-convex objectives with specific gradient patterns. In practice, AdamW inherits this theoretical uncertainty. There are also tasks, particularly in computer vision, where carefully tuned SGD with momentum can match or outperform AdamW. The advantage of AdamW over Adam is specifically in regularization quality, not in convergence guarantees or optimization dynamics more broadly.

The adaptive learning rate mechanism in AdamW, inherited directly from Adam, also means that the effective learning rate per parameter is not transparent. Understanding what the optimizer is actually doing to a specific parameter at a specific training step requires knowing its full gradient history, which is not readily available during training. This opacity makes hyperparameter debugging harder: when a model behaves unexpectedly during training, it can be difficult to determine whether the issue lies in the learning rate, the weight decay, the beta parameters, or some interaction among them.

The impact of AdamW on language model training extends beyond the mathematical correction. Language models have exactly the parameter structure where the Adam+L2 flaw is most harmful: a vocabulary of tens or hundreds of thousands of tokens, each with its own embedding, updated with wildly different frequencies over the course of training. Common tokens in a large corpus (function words like "the", "of", "a") might appear billions of times across training data, while rare scientific or technical terms might appear only thousands of times. The ratio of gradient history magnitudes across these extremes can easily exceed 1000 to 1, meaning Adam+L2 would apply regularization that is 30 times stronger to the rare token embeddings than to the common ones.

Beyond embeddings, attention weight matrices in transformer models receive gradients that vary substantially across heads and layers. Earlier layers in a deep transformer tend to capture more syntactic, general patterns and may receive more consistent, moderate gradients. Later layers often specialize in task-specific patterns and can receive more varied, sometimes sparser gradient signals. This heterogeneity means the Adam+L2 flaw affects embeddings and the broader parameter space of a transformer.

Loshchilov and Hutter's experiments showed consistent improvements when switching from Adam+L2 to AdamW on language modeling tasks. Subsequent work across many groups confirmed these findings, particularly in regimes where regularization is most important: pretraining with limited compute (where overfitting to the finite training set is a real risk), fine-tuning on small datasets, and training large models that would otherwise show training instabilities from unconstrained weight growth.

The LLM training community adopted AdamW quickly. GPT-2 (2019), GPT-3 (2020), BERT variants, and essentially all major language models released after 2018 use AdamW. It has become one of those foundational defaults that practitioners apply automatically because the alternative has a documented flaw.

The insight behind AdamW, that regularization should be applied to the parameters directly rather than mixed into the gradient, generalizes beyond this specific fix. The more general principle is that different aspects of the optimization problem should be kept separated when they have different computational properties. Gradient-based optimization and regularization-based constraint satisfaction are fundamentally different operations, and conflating them through the gradient produces confounded behavior that depends on the optimizer's internal scaling.

This principle has influenced subsequent optimizer research. The AdaFactor optimizer, designed for memory efficiency, decouples its factorized second moment estimate from the regularization term. The Lion optimizer, which uses the sign of the gradient rather than the gradient magnitude, does not even have a natural way to incorporate L2 regularization via the gradient, making decoupled weight decay a necessity rather than a choice. In general, as optimizers become more sophisticated in their gradient processing, the argument for keeping regularization separate grows stronger.

AdamW requires storing three buffers per parameter: the parameter itself, the first moment estimate, and the second moment estimate. For a 7 billion parameter model with float32 precision, this is approximately 84 GB just for the optimizer state, in addition to the 28 GB for the model parameters themselves. This memory overhead is one of the main practical challenges in training large language models and has motivated research into memory-efficient alternatives.

AdaFactor addresses this by approximating the second moment matrix with a low-rank factorization, dramatically reducing memory at the cost of some optimization quality. 8-bit Adam and related quantized optimizers reduce memory by storing the moment estimates in lower precision. Some training recipes use Lion or other sign-based optimizers that eliminate the second moment entirely. Despite these alternatives, AdamW remains the standard for most LLM training, particularly at scales where the memory overhead is manageable.

AdamW is not the final word in optimizer research. Several alternatives have been proposed and have shown promise in specific settings:

Adan (Adaptive Nesterov Momentum Algorithm) extends the AdamW framework with a Nesterov-style momentum that estimates the gradient at the anticipated future parameter position rather than the current one. This provides a more aggressive form of momentum that can improve convergence speed on some tasks. Adan has shown competitive results on image classification and some language modeling benchmarks.

Lion (Evolved Sign Momentum), proposed by Google Brain in 2023, uses the sign of a momentum-like quantity rather than the gradient magnitude for parameter updates. This produces uniform-magnitude updates across all parameters, with lower memory usage than AdamW (no second moment needed). Lion has shown competitive performance on some language model pretraining tasks at moderate scales, though it requires different hyperparameter settings and its behavior can be harder to reason about than AdamW.

AdaFactor reduces memory usage by factorizing the second moment matrix into its row and column factors, making it feasible to train very large models without the full Adam moment buffer. AdaFactor was used for training some versions of T5 and related models at Google. The memory savings come at some optimization quality cost, and it requires more careful hyperparameter tuning than AdamW.

Sophia uses second-order curvature information (specifically, a diagonal approximation of the Hessian) to scale updates more precisely than Adam's gradient-based variance estimate. The idea is that the Hessian curvature provides a better signal for step size than gradient history alone. Sophia showed faster loss reduction per training step in some large language model experiments, at the cost of additional computation to estimate the Hessian.

Muon (Momentum Orthogonalized by Newton-schulz) applies an orthogonalization step to the gradient before computing the update, which can improve the conditioning of the optimization problem. Muon has attracted attention for its performance on small-to-medium scale language models.

Despite these alternatives, AdamW remains the dominant choice for large language model training as of 2025. Its stability, well-understood behavior, extensive community support, and the massive amount of infrastructure built around it make switching to an alternative a significant undertaking that requires careful validation. Newer optimizers have shown promise in research settings, but the bar for displacing AdamW in production LLM training is high.