AdamW: Decoupled Weight Decay for Neural Networks

Adam became the default optimizer for deep learning almost immediately after its 2014 publication. It combines momentum with adaptive learning rates, and for most tasks, it simply works. Practitioners adopted it widely, and it quickly became the go-to choice for training language models, image classifiers, and a range of other architectures. Yet there is a subtle flaw in how Adam handles regularization, one that went largely unnoticed for three years until Ilya Loshchilov and Frank Hutter pointed it out in their 2017 paper "Decoupled Weight Decay Regularization." The fix they proposed, AdamW, is now the standard optimizer for training large language models.

The flaw is not an edge case or a minor numerical issue. It is a fundamental mathematical inconsistency between what practitioners intend when they set a "weight decay" parameter and what Adam actually computes. Understanding this inconsistency requires stepping back from the optimizer itself and examining what regularization is trying to do in the first place, and then tracing exactly how Adam distorts that intent.

This chapter walks through that complete arc, from the motivation for regularization in neural networks to the precise mathematical analysis of why Adam handles it incorrectly, through the AdamW formulation and its practical implications for LLM training. By the end, you will have a solid understanding of how to use AdamW, why it was necessary, and what it achieves.

When training a neural network, the optimization objective is to minimize the loss on training data. But the true goal is generalization: you want the model to perform well on new, unseen data. These objectives can conflict. A model with enough capacity can memorize training examples perfectly while learning nothing transferable about the underlying patterns.

The formal version of this tension is the bias-variance tradeoff. A model with too few parameters underfits: it cannot capture the true patterns in the data. A model with too many parameters, trained without constraints, overfits: it learns the training data including its noise, producing predictions that are precise on training examples but poor on new ones. Regularization techniques introduce controlled constraints that push the model toward simpler solutions, reducing the risk of overfitting without unnecessarily limiting its capacity.

The connection between weight magnitude and generalization is intuitive once you think about what large weights imply. A neural network with very large weights is one where small changes in the input produce large changes in the output. This sensitivity to input variations means the network is capable of memorizing fine-grained patterns in the training data, including noise. A network with small weights, by contrast, produces outputs that change smoothly with inputs, which corresponds to learning more general patterns rather than specific examples.

This intuition underlies L2 regularization, which applies a direct penalty to large weights. The idea is that among all solutions that fit the training data approximately equally well, we prefer the one with the smallest weights. This preference is a prior belief about the structure of the problem: we expect the true underlying function to be smooth and not reliant on any individual input feature having overwhelming influence.

There is also a Bayesian interpretation. If you place a zero-centered Gaussian prior on the weights, $p(\theta) \propto \exp(-\lambda \|\theta\|^2)$, then maximum a posteriori (MAP) inference under this prior corresponds exactly to training with L2 regularization. The regularization coefficient $\lambda$ controls the strength of this prior, effectively saying how much you trust your prior belief about small weights versus the evidence from the data.

In practice, regularization helps most when the model is large relative to the training data, when the training data is noisy or biased, or when you intend to fine-tune the model on a small later dataset. For large language models, all three of these conditions often apply simultaneously, which is why regularization plays an large effect in LLM training despite the enormous dataset sizes involved.

Understanding the distinction between L2 regularization and weight decay is the core insight behind AdamW. These two approaches address the same goal from different angles, and for a long time, practitioners used the terms interchangeably. They are identical in the context of standard gradient descent, but they diverge for adaptive optimizers, and that divergence has significant practical consequences.

L2 regularization adds a penalty term to the training loss. The modified objective function becomes:

where:

• $L(\theta)$: the original training loss (cross-entropy, MSE, etc.)
• $\lambda$: the regularization coefficient controlling penalty strength
• $\|\theta\|^2 = \sum_i \theta_i^2$: the squared L2 norm of all parameters
• $\tilde{L}(\theta)$: the regularized loss that the optimizer minimizes

When you compute the gradient of $\tilde{L}$ with respect to a parameter $\theta_i$, the regularization term contributes an extra additive term:

The gradient now has two components: the original loss gradient, which points toward lower training loss, plus $\lambda \theta_i$, which always pushes $\theta_i$ toward zero. The optimizer sees a single combined gradient, with no distinction between the signal from data and the signal from regularization. The optimizer does not know that part of the gradient it receives is a regularization term rather than a data-derived signal. It simply processes the combined gradient as if it were all signal from the loss.

Weight decay is a different approach to the same goal. Instead of modifying the loss function, weight decay modifies the parameter update directly. After computing the gradient update from the loss, you shrink the weights by a fixed fraction:

where:

• $\theta_t$: the parameter value at step $t$
• $\lambda$: the weight decay coefficient
• $\alpha$: the learning rate
• $\nabla_{\theta} L(\theta_t)$: the gradient of the training loss

The weight decay term $(1-\lambda)\theta_t$ multiplies the current parameter value by a number slightly less than 1, which shrinks it toward zero. This shrinkage is completely independent of the gradient. It happens as a direct multiplicative operation on the parameter value, regardless of what the gradient is doing or how large recent gradients have been. The gradient step and the weight decay step are conceptually separate operations, even though they are combined into a single update equation.

For standard stochastic gradient descent with learning rate $\alpha$, the two approaches produce mathematically identical updates. The SGD update with L2 regularization is:

Compare this to the weight decay update: $\theta_{t+1} = (1 - \lambda') \theta_t - \alpha \nabla_{\theta} L(\theta_t)$.

These are identical when $\lambda' = \alpha \lambda$. You can translate between the two by rescaling the regularization coefficient by the learning rate. For SGD, L2 regularization and weight decay are equivalent formulations of the same thing, just parameterized differently.

This equivalence is why many practitioners and many codebases used the terms interchangeably. Weight decay in PyTorch's SGD optimizer actually implements L2 regularization via the gradient, and for SGD it makes no difference. The naming inconsistency in the deep learning ecosystem is a direct consequence of this historical equivalence being assumed to hold universally, when in fact it breaks down as soon as you move beyond SGD.

Even though L2 regularization and weight decay are equivalent for SGD, they reflect different computational philosophies. L2 regularization thinks about regularization as part of the objective: you are literally optimizing a different function, one that includes a preference for small weights. Weight decay thinks about regularization as part of the update rule: you are adding a direct shrinkage operation that is applied after the gradient step.

For simple first-order optimizers, this distinction does not matter because the gradient step scales uniformly across all parameters. But once you introduce any form of adaptive scaling, the two approaches diverge because the adaptive scaling modifies the gradient-based update but not the direct parameter update.

Adam is an adaptive optimizer: it scales each parameter's gradient by a factor that depends on the history of squared gradients for that parameter. This is Adam's greatest strength, allowing it to effectively handle parameters with very different gradient scales. But this same adaptive scaling is precisely what breaks the equivalence between L2 regularization and weight decay.

Recall from the previous chapter that Adam maintains two moment estimates and applies them to scale the gradient:

where $\hat{m}_t$ is the bias-corrected first moment (a smoothed version of the current gradient) and $\hat{v}_t$ is the bias-corrected second moment (a smoothed version of the squared gradient, capturing gradient magnitude history).

The key property is that $\hat{v}_t$ differs across parameters. A parameter that has historically received large gradients has a large $\hat{v}_t$, which shrinks its effective learning rate. A parameter that has received small gradients has a small $\hat{v}_t$, which gives it a larger effective learning rate. This per-parameter adaptive scaling is what lets Adam handle heterogeneous gradient distributions.

Now, what happens when you apply L2 regularization with Adam? The combined gradient passed to Adam includes the regularization term:

where $g_t = \nabla_{\theta} L(\theta_t)$ is the original gradient. Adam then uses this regularized gradient to update both moment estimates and the parameters:

The adaptive scaling term $\hat{v}_t$ now accumulates the regularization signal $\lambda \theta_t$ along with the loss gradient. The effective learning rate applied to the regularization penalty is:

This effective rate differs from parameter to parameter depending on the history of squared gradients. Parameters that have received large gradients from the loss will have large $\hat{v}_t$ values, which reduces their effective learning rate, including the rate at which regularization shrinks them. Parameters with small historical gradients have small $\hat{v}_t$ values, giving them a larger effective learning rate, and consequently stronger effective regularization.

The result is that L2 regularization does not apply uniformly across parameters when used with Adam. The effective regularization strength for parameter $i$ is roughly:

This effective strength is high for rarely-updated parameters (small $\hat{v}_{t,i}$) and low for frequently-updated parameters (large $\hat{v}_{t,i}$). In a language model, word embeddings for common tokens receive dense, large gradient updates, giving them large $\hat{v}$ values and therefore weak effective regularization. Embeddings for rare tokens receive sparse, small updates, giving them small $\hat{v}$ values and therefore strong effective regularization.

This is the opposite of what good regularization should do. High-frequency embeddings, which are updated on almost every training step and are most at risk of overfitting to the training data distribution, receive the weakest regularization. Low-frequency embeddings, which are rarely updated and represent underspecified representations, receive the strongest regularization. The regularization strength has become an artifact of gradient history rather than a deliberate design choice.

This is the observation Loshchilov and Hutter formalized in their 2017 paper "Decoupled Weight Decay Regularization." Their analysis had two parts: a theoretical derivation showing why L2 regularization and weight decay are not equivalent for adaptive optimizers, and empirical experiments on image classification and language modeling tasks showing that the decoupled version outperforms the coupled version. They found that AdamW outperforms Adam+L2 on a variety of benchmarks when both are carefully tuned, and that the improvement is more pronounced as regularization becomes more important, specifically in regimes with smaller datasets relative to model size, or with more aggressive regularization strengths.

An important nuance in their analysis: the optimal hyperparameters for Adam+L2 and AdamW are not interchangeable. Because Adam+L2 applies non-uniform effective regularization, a weight_decay value that seems to work in Adam is not numerically equivalent to the same value in AdamW. The Loshchilov-Hutter paper showed that the optimal weight decay for AdamW tends to be larger in absolute terms, because the effective regularization per parameter is more predictable and you can apply stronger regularization without the unintended side effect of over-regularizing rarely-updated parameters.

AdamW fixes the problem by applying weight decay directly to the parameters, separate from the gradient update. The regularization term never enters the moment estimates. The moments track only the gradient signal from the loss, and the weight decay operates independently as a direct multiplicative shrinkage of the parameters.

The standard Adam update with L2 regularization computes:

where the moments $m_t$ and $v_t$ are computed from the regularized gradient $g_t + \lambda \theta_t$.

The AdamW update decouples these components:

where:

• $\hat{m}_t$: bias-corrected first moment, computed from the original gradient $g_t$ only
• $\hat{v}_t$: bias-corrected second moment, computed from the original gradient $g_t$ only
• $\lambda$: the weight decay coefficient
• $\alpha$: the learning rate

The weight decay term $\alpha \lambda \theta_t$ is added outside the adaptive scaling. Every parameter is shrunk by the same multiplicative factor $\alpha \lambda$ at each step, regardless of its gradient history. The adaptive learning rates only affect the gradient-based update; they have no influence on how strongly weight decay pulls each parameter toward zero.

Written out as the complete set of equations, AdamW performs these steps at each training iteration:

where:

• $g_t$: gradient of the training loss, with no regularization term mixed in
• $m_t$: first moment estimate, an exponential moving average of gradients
• $v_t$: second moment estimate, an exponential moving average of squared gradients
• $\hat{m}_t$: bias-corrected first moment, compensating for the zero initialization of $m_0$
• $\hat{v}_t$: bias-corrected second moment, compensating for the zero initialization of $v_0$
• $\beta_1$: first moment decay rate, typically 0.9
• $\beta_2$: second moment decay rate, typically 0.999
• $\epsilon$: numerical stability constant, typically 1e-8
• $\lambda$: weight decay coefficient
• $\alpha$: learning rate

The critical difference from Adam with L2 regularization is in the last equation. The weight decay applies uniformly: the same $\alpha \lambda \theta_{t-1}$ term acts on every parameter with the same effective coefficient, while the gradient update $\hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$ handles the adaptive per-parameter scaling independently.

Another way to think about the difference is geometric. Adam with L2 regularization is trying to minimize a particular objective function, namely the regularized loss $\tilde{L}(\theta)$, but using a preconditioned gradient descent step where the preconditioner is the adaptive scaling $1/\sqrt{\hat{v}_t}$. The preconditioning changes the effective geometry of the optimization landscape, and the L2 ball that L2 regularization defines in the original parameter space gets distorted by the preconditioner.

AdamW is doing something subtly different: it separates the geometry-distorting adaptive step for the loss gradient from the direct parameter shrinkage for weight decay. The adaptive step handles the optimization of the loss; the weight decay handles the regularization constraint. These two objectives are kept separate rather than being mixed into a single gradient computation.

This is also why AdamW has a cleaner connection to the original weight decay formulation from SGD literature. In the SGD with momentum world, weight decay was a direct multiplicative shrinkage applied to parameters. AdamW restores this interpretation in the adaptive optimizer setting, where Adam's adaptive scaling makes it impossible to achieve uniform weight decay through gradient modification alone.

To see the difference concretely, consider two parameters in a language model:

• Parameter A: an embedding for a common word, frequently updated with large gradients. Suppose $\hat{v}_t = 0.01$ (reflecting a large gradient history).
• Parameter B: an embedding for a rare word, infrequently updated with small gradients. Suppose $\hat{v}_t = 0.0001$ (reflecting a small gradient history).

With L2 regularization in Adam (coupling), the effective regularization multiplier for each parameter is roughly $\lambda / (\sqrt{\hat{v}_t} + \epsilon)$. Setting $\lambda = 0.01$ and $\epsilon = 10^{-8}$:

• Parameter A (common word): effective regularization $\approx 0.01 / \sqrt{0.01} = 0.01 / 0.1 = 0.1$
• Parameter B (rare word): effective regularization $\approx 0.01 / \sqrt{0.0001} = 0.01 / 0.01 = 1.0$

Parameter B experiences ten times stronger effective regularization than parameter A, simply because it has a smaller gradient history. But consider what this means in practice. The common word embedding has been updated on nearly every training step and has had the most opportunity to overfit to training data. The rare word embedding has been updated infrequently and is likely underdetermined, not overfitted. Adam+L2 applies strong regularization to the underdetermined parameter and weak regularization to the potentially overfitted one: exactly backwards from what you would want.

With AdamW (decoupled weight decay), both parameters receive the same $\alpha \lambda = \alpha \times 0.01$ decay factor at every step. Parameter A and parameter B are both shrunk by the same multiplicative factor, regardless of how often they have been updated or how large their historical gradients have been. The regularization is uniform, as intended.

This concrete example illustrates why AdamW matters particularly for language models, which have exactly this structure: a vocabulary of many tokens with wildly varying frequencies, producing embedding gradients with highly heterogeneous statistics.

Let's implement both Adam with L2 regularization and AdamW from scratch to see the difference in practice. We'll train a small regression network on noisy data and compare how the two approaches handle regularization.

The weight norm comparison reveals a key behavioral difference. AdamW achieves more controlled weight norms because the decay is applied uniformly without interference from the adaptive learning rate scaling. Adam+L2's weight norms are less predictable because the effective regularization strength varies by parameter, and the optimizer is not actually minimizing the same objective you think it is.

Another way to understand the difference is to look at how the effective regularization strength varies across parameters with different gradient histories. This is the analytical picture underlying the numerical example from earlier.

The plot makes the problem concrete. For parameters with small gradient histories (left side of the x-axis), Adam+L2 applies extremely strong effective regularization, up to 100 times stronger than for frequently-updated parameters (right side). AdamW's flat line demonstrates its key advantage: the regularization coefficient has a consistent, predictable meaning across all parameters. When you set weight_decay=0.01 in AdamW, every single parameter in the model is shrunk by a factor of $0.01 \times \alpha$ at each step, no more and no less.

In practice, you will use PyTorch's implementation rather than writing your own. PyTorch provides torch.optim.AdamW with a clean API that correctly implements the decoupled weight decay.

This shows a standard PyTorch training loop with AdamW. The weight_decay parameter directly corresponds to $\lambda$ in the AdamW formulation. Unlike PyTorch's torch.optim.Adam optimizer where weight_decay adds L2 regularization to the gradient, torch.optim.AdamW correctly implements decoupled weight decay. Note that PyTorch's Adam optimizer documentation explicitly warns about this distinction and recommends AdamW when you want proper weight decay behavior.

One practical advantage of AdamW's decoupled formulation is that you can apply different weight decay rates to different parts of the model in a principled way. In transformer models, it is common practice to apply weight decay to weight matrices but not to bias terms or layer normalization parameters. This pattern was used in the original BERT paper, GPT-2, GPT-3, and has become a near-universal convention in LLM training.

This pattern of separating weight matrices from biases and normalization parameters is standard in large language model training. The intuition is that biases and normalization scale factors serve different functional roles than weight matrices. A bias term shifts the activation function input; shrinking it toward zero would force the model to use zero-centered activations regardless of what the data requires. Layer normalization parameters control the scale and offset of normalized activations; decaying them toward zero could interfere with the normalization's intended effect.

Weight matrices, by contrast, encode learned transformations between representation spaces. Keeping these matrices from growing too large is a sensible regularization goal because large weight matrix norms are the direct mechanism by which a model can become overly sensitive to specific input patterns. The parameter group pattern lets you apply weight decay precisely where it helps and skip it where it might hurt.

The key AdamW hyperparameters and their practical interpretation are:

• lr (learning rate): Controls the overall step size. Typical values range from 1e-4 to 3e-4 for large language models during pretraining. A warmup schedule is always used, linearly increasing the learning rate from 0 over the first several thousand steps before following a cosine or linear decay to the end of training.
• betas=(beta1, beta2): Decay rates for the first and second moment estimates. The default (0.9, 0.999) works well for most tasks. Some LLM training recipes use beta2=0.95 or beta2=0.98 to allow faster adaptation to gradient magnitude changes over the course of training.
• eps: Numerical stability constant. The default 1e-8 is appropriate for float32 training. When training in float16 or bfloat16, increasing to 1e-6 or 1e-7 can improve numerical stability by reducing the impact of floating-point rounding on the denominator $\sqrt{\hat{v}_t} + \epsilon$.
• weight_decay: The decoupled weight decay coefficient $\lambda$. Typical values are 0.01 to 0.1 for language model pretraining, with 0.01 common for fine-tuning where strong regularization can impair task-specific adaptation.

The difference between Adam+L2 and AdamW is most visible when regularization matters most, specifically at larger regularization strengths and in models with heterogeneous gradient distributions. Let's run a systematic comparison across different weight decay values.

The plot reveals how Adam+L2 and AdamW respond differently to increasing regularization. AdamW shows more consistent and predictable weight decay behavior: as you increase the regularization coefficient, the final weight norm decreases correspondingly. Adam+L2 shows a weaker response to increasing regularization, particularly at higher values, because the adaptive scaling is partially counteracting the intended effect of the L2 penalty for parameters with large gradient histories.

AdamW is the default optimizer for virtually all large language model training pipelines. The specific hyperparameters vary by model size, architecture, and training recipe, but several values have emerged as reliable defaults through extensive empirical validation across many research groups and institutions.

The learning rate is the most sensitive hyperparameter in LLM training. Choosing a value that is too large causes instability early in training, where gradient norms spike unpredictably and the model may diverge entirely. A value that is too small leads to slow convergence and underutilization of the available compute budget.

A warmup phase is almost universally used. During warmup, the learning rate is increased linearly from 0 (or a very small value) over the first 1-5% of training steps. The purpose of warmup is to allow the moment estimates in AdamW to stabilize before the optimizer is making large parameter updates. At the start of training, the second moment estimate $v_t$ is initialized to zero and slowly accumulates information about the gradient distribution. If the learning rate is large from the very beginning, the denominator $\sqrt{\hat{v}_t} + \epsilon$ is initially dominated by $\epsilon$, which means the effective learning rate is much larger than $\alpha$. Warmup prevents this by keeping the learning rate small during the period when the moment estimates are unreliable.

After warmup, a decay schedule is applied. Cosine decay is the most common choice, smoothly reducing the learning rate to near zero over the remainder of training. Linear decay is also used. Some recipes use a constant learning rate after warmup with a final short decay phase. The choice of schedule affects both the total training cost and the quality of the final model.

The original BERT training recipe established a widely used set of defaults for encoder-style models:

• Learning rate: 1e-4 to 5e-4, with linear warmup over the first 1% of steps, then linear decay to zero
• Weight decay: 0.01
• Beta1: 0.9, Beta2: 0.999
• Gradient clipping: max norm of 1.0
• No weight decay on bias terms or LayerNorm parameters

These values have been validated across many BERT variants and related encoder models. The relatively small weight decay of 0.01 is appropriate for large-scale pretraining, where the dataset is large enough that overfitting is not the primary concern and regularization's main role is to keep weights bounded rather than to prevent memorization.

Decoder-style autoregressive models use somewhat different hyperparameters. The original GPT-3 training recipe, described in the Brown et al. 2020 paper, used:

• Learning rate: 6e-5 for the 175B parameter model, with cosine decay and warmup over 375 million tokens
• Weight decay: 0.1
• Beta1: 0.9, Beta2: 0.95
• Gradient clipping: max norm of 1.0
• No weight decay on bias terms

The choice of Beta2=0.95 rather than 0.999 is notable. A smaller Beta2 gives the second moment estimate a shorter effective memory window, allowing the optimizer to respond more quickly to changes in the gradient distribution as training progresses. For very large models trained on diverse datasets over a long period, the gradient statistics can shift significantly across training phases, and a more responsive second moment estimate can be beneficial. Some subsequent work has found that Beta2 between 0.95 and 0.99 works well for large decoder models, while 0.999 remains fine for smaller models or shorter training runs.

The higher weight decay value of 0.1 in GPT-3 compared to BERT's 0.01 reflects a deliberate choice to apply stronger regularization for large-scale decoder models. At the scale of hundreds of billions of parameters, the risk of the model developing degenerate solutions or excessively large weights in certain components is real, and stronger weight decay provides a useful stabilizing constraint.

When fine-tuning a pretrained model on a smaller downstream dataset, the hyperparameter landscape shifts:

• Learning rate: 1e-5 to 5e-5, often with a short linear warmup over the first few hundred steps
• Weight decay: 0.01 or lower
• Beta1: 0.9, Beta2: 0.999
• Fewer total training steps, shorter decay schedule

Fine-tuning benefits from smaller learning rates because the model starts from a good pretrained initialization and needs precise, controlled adaptation to the target task. Large learning rates during fine-tuning can catastrophically destroy the pretrained representations, causing the model to lose general language understanding while adapting to the specific task. Weight decay is also reduced during fine-tuning: on small datasets, strong regularization can prevent the model from fully adapting to the target task, degrading performance on the very task you are trying to improve.

Because AdamW applies weight decay as $\alpha \lambda \theta$, the learning rate and weight decay are not independent. The effective weight decay per step is $\alpha \lambda$, not just $\lambda$. When you change the learning rate, the effective weight decay changes proportionally. Some practitioners treat the ratio $\lambda / \alpha$ as a stable hyperparameter and rescale weight decay whenever they change the learning rate, though this is a heuristic rather than a rigorously derived rule.

This coupling between learning rate and weight decay also matters when using learning rate schedules. Under a cosine decay schedule, the learning rate decreases to near zero at the end of training, which means the effective weight decay also decreases to near zero. In the final stages of training under cosine decay, the model is receiving almost no regularization. This is sometimes intentional: late-stage training is about fine-tuning the solution rather than constraining it, and reducing weight decay at this stage can allow the model to converge more cleanly to its optimal point without being pulled away by regularization.

One of the most practically important scenarios for AdamW is language model embedding training. Embeddings for common tokens receive dense updates while embeddings for rare tokens receive sparse updates. Let's simulate this embedding scenario to see the difference directly.

This visualization captures exactly why AdamW matters for language model training. Under Adam+L2, common token embeddings receive weak regularization and rare token embeddings receive strong regularization. The regularization strength is driven by the training corpus statistics rather than by any deliberate hyperparameter choice. Under AdamW, the flat line shows that every embedding receives exactly the regularization you intended when you set weight_decay=0.01.

AdamW is a meaningful improvement over Adam with L2 regularization, but it inherits many of Adam's properties and does not solve all optimization challenges. Understanding where AdamW helps, where it does not, and why it became the standard for language model training gives you a complete picture of its role in the deep learning toolkit.

AdamW corrects the mathematical inconsistency of L2 regularization inside Adam, making the weight decay hyperparameter behave as practitioners intend. This is the fundamental contribution of the Loshchilov-Hutter paper: not a dramatic algorithmic innovation, but a precise identification of an existing inconsistency and a minimal fix that restores the intended behavior.

What AdamW does not fix is the general challenge of tuning regularization in deep learning. The optimal weight decay coefficient still requires tuning, and its interaction with the learning rate schedule adds complexity. A cosine decay schedule effectively reduces weight decay's influence in later training stages as the learning rate decreases, which can be desirable (allowing the model to converge cleanly at the end) or undesirable depending on the training regime. There is no automatic way to set weight decay; it remains an important hyperparameter that requires experimental validation.

AdamW also does not resolve some of Adam's other known limitations. There exist convergence results showing that Adam can converge to suboptimal solutions in certain settings, particularly on non-convex objectives with specific gradient patterns. In practice, AdamW inherits this theoretical uncertainty. There are also tasks, particularly in computer vision, where carefully tuned SGD with momentum can match or outperform AdamW. The advantage of AdamW over Adam is specifically in regularization quality, not in convergence guarantees or optimization dynamics more broadly.

The adaptive learning rate mechanism in AdamW, inherited directly from Adam, also means that the effective learning rate per parameter is not transparent. Understanding what the optimizer is actually doing to a specific parameter at a specific training step requires knowing its full gradient history, which is not readily available during training. This opacity makes hyperparameter debugging harder: when a model behaves unexpectedly during training, it can be difficult to determine whether the issue lies in the learning rate, the weight decay, the beta parameters, or some interaction among them.

The impact of AdamW on language model training extends beyond the mathematical correction. Language models have exactly the parameter structure where the Adam+L2 flaw is most harmful: a vocabulary of tens or hundreds of thousands of tokens, each with its own embedding, updated with wildly different frequencies over the course of training. Common tokens in a large corpus (function words like "the", "of", "a") might appear billions of times across training data, while rare scientific or technical terms might appear only thousands of times. The ratio of gradient history magnitudes across these extremes can easily exceed 1000 to 1, meaning Adam+L2 would apply regularization that is 30 times stronger to the rare token embeddings than to the common ones.

Beyond embeddings, attention weight matrices in transformer models receive gradients that vary substantially across heads and layers. Earlier layers in a deep transformer tend to capture more syntactic, general patterns and may receive more consistent, moderate gradients. Later layers often specialize in task-specific patterns and can receive more varied, sometimes sparser gradient signals. This heterogeneity means the Adam+L2 flaw affects embeddings and the broader parameter space of a transformer.

Loshchilov and Hutter's experiments showed consistent improvements when switching from Adam+L2 to AdamW on language modeling tasks. Subsequent work across many groups confirmed these findings, particularly in regimes where regularization is most important: pretraining with limited compute (where overfitting to the finite training set is a real risk), fine-tuning on small datasets, and training large models that would otherwise show training instabilities from unconstrained weight growth.

The LLM training community adopted AdamW quickly. GPT-2 (2019), GPT-3 (2020), BERT variants, and essentially all major language models released after 2018 use AdamW. It has become one of those foundational defaults that practitioners apply automatically because the alternative has a documented flaw.

The insight behind AdamW, that regularization should be applied to the parameters directly rather than mixed into the gradient, generalizes beyond this specific fix. The more general principle is that different aspects of the optimization problem should be kept separated when they have different computational properties. Gradient-based optimization and regularization-based constraint satisfaction are fundamentally different operations, and conflating them through the gradient produces confounded behavior that depends on the optimizer's internal scaling.

This principle has influenced subsequent optimizer research. The AdaFactor optimizer, designed for memory efficiency, decouples its factorized second moment estimate from the regularization term. The Lion optimizer, which uses the sign of the gradient rather than the gradient magnitude, does not even have a natural way to incorporate L2 regularization via the gradient, making decoupled weight decay a necessity rather than a choice. In general, as optimizers become more sophisticated in their gradient processing, the argument for keeping regularization separate grows stronger.

AdamW requires storing three buffers per parameter: the parameter itself, the first moment estimate, and the second moment estimate. For a 7 billion parameter model with float32 precision, this is approximately 84 GB just for the optimizer state, in addition to the 28 GB for the model parameters themselves. This memory overhead is one of the main practical challenges in training large language models and has motivated research into memory-efficient alternatives.

AdaFactor addresses this by approximating the second moment matrix with a low-rank factorization, dramatically reducing memory at the cost of some optimization quality. 8-bit Adam and related quantized optimizers reduce memory by storing the moment estimates in lower precision. Some training recipes use Lion or other sign-based optimizers that eliminate the second moment entirely. Despite these alternatives, AdamW remains the standard for most LLM training, particularly at scales where the memory overhead is manageable.

AdamW is not the final word in optimizer research. Several alternatives have been proposed and have shown promise in specific settings:

Adan (Adaptive Nesterov Momentum Algorithm) extends the AdamW framework with a Nesterov-style momentum that estimates the gradient at the anticipated future parameter position rather than the current one. This provides a more aggressive form of momentum that can improve convergence speed on some tasks. Adan has shown competitive results on image classification and some language modeling benchmarks.

Lion (Evolved Sign Momentum), proposed by Google Brain in 2023, uses the sign of a momentum-like quantity rather than the gradient magnitude for parameter updates. This produces uniform-magnitude updates across all parameters, with lower memory usage than AdamW (no second moment needed). Lion has shown competitive performance on some language model pretraining tasks at moderate scales, though it requires different hyperparameter settings and its behavior can be harder to reason about than AdamW.

AdaFactor reduces memory usage by factorizing the second moment matrix into its row and column factors, making it feasible to train very large models without the full Adam moment buffer. AdaFactor was used for training some versions of T5 and related models at Google. The memory savings come at some optimization quality cost, and it requires more careful hyperparameter tuning than AdamW.

Sophia uses second-order curvature information (specifically, a diagonal approximation of the Hessian) to scale updates more precisely than Adam's gradient-based variance estimate. The idea is that the Hessian curvature provides a better signal for step size than gradient history alone. Sophia showed faster loss reduction per training step in some large language model experiments, at the cost of additional computation to estimate the Hessian.

Muon (Momentum Orthogonalized by Newton-schulz) applies an orthogonalization step to the gradient before computing the update, which can improve the conditioning of the optimization problem. Muon has attracted attention for its performance on small-to-medium scale language models.

Despite these alternatives, AdamW remains the dominant choice for large language model training as of 2025. Its stability, well-understood behavior, extensive community support, and the massive amount of infrastructure built around it make switching to an alternative a significant undertaking that requires careful validation. Newer optimizers have shown promise in research settings, but the bar for displacing AdamW in production LLM training is high.

AdamW corrects a subtle but consequential flaw in how standard Adam handles regularization. The key ideas are:

• L2 regularization and weight decay are equivalent for SGD, because SGD applies a uniform learning rate to all parameters. The regularization term in the gradient gets the same scaling as the loss gradient, so the two formulations produce identical parameter updates up to a rescaling of the coefficient.
• They diverge for adaptive optimizers like Adam, because Adam's per-parameter adaptive scaling applies different effective learning rates to different parameters. When the regularization penalty is mixed into the gradient, it gets scaled differently for each parameter based on that parameter's gradient history. Parameters with large historical gradients receive weaker effective regularization than parameters with small historical gradients.
• In language models, this matters most for embeddings, where common token embeddings receive dense, large gradients and thus weak effective regularization under Adam+L2, while rare token embeddings receive sparse, small gradients and thus strong effective regularization. This is backwards: common tokens, updated most often, are the most at risk of overfitting.
• AdamW fixes this by decoupling: the weight decay is applied directly to the parameters as $\alpha \lambda \theta_t$, outside the adaptive scaling term. Every parameter receives the same weight decay factor at each step, and the regularization coefficient $\lambda$ has a consistent, interpretable meaning independent of gradient history.
• The hyperparameter semantics change: a weight_decay value in AdamW behaves predictably, while the same value in standard Adam does not. Optimal weight decay values for AdamW tend to be larger than what was used with Adam+L2 because the effective regularization per parameter is more stable.
• Standard practice in LLM training uses AdamW with weight_decay=0.01 to 0.1, learning rate 1e-4 to 3e-4 with warmup and cosine decay, and bias terms plus LayerNorm parameters excluded from weight decay.
• AdamW is the default optimizer for LLM training: GPT-2, GPT-3, BERT, and essentially all major language models since 2018 use AdamW, and it remains the dominant choice for large-scale training as of 2025.

The next chapter covers weight initialization, the practice of setting parameter starting values before training begins. Poor initialization can cause gradients to vanish or explode even before the first update, making choices like Xavier and He initialization essential for training deep networks reliably.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about AdamW and decoupled weight decay.