AdamW › Typical Hyperparameter Values for LLM Training

AdamW is the default optimizer for virtually all large language model training pipelines. The specific hyperparameters vary by model size, architecture, and training recipe, but several values have emerged as reliable defaults through extensive empirical validation across many research groups and institutions.

The learning rate is the most sensitive hyperparameter in LLM training. Choosing a value that is too large causes instability early in training, where gradient norms spike unpredictably and the model may diverge entirely. A value that is too small leads to slow convergence and underutilization of the available compute budget.

A warmup phase is almost universally used. During warmup, the learning rate is increased linearly from 0 (or a very small value) over the first 1-5% of training steps. The purpose of warmup is to allow the moment estimates in AdamW to stabilize before the optimizer is making large parameter updates. At the start of training, the second moment estimate $v_t$ is initialized to zero and slowly accumulates information about the gradient distribution. If the learning rate is large from the very beginning, the denominator $\sqrt{\hat{v}_t} + \epsilon$ is initially dominated by $\epsilon$, which means the effective learning rate is much larger than $\alpha$. Warmup prevents this by keeping the learning rate small during the period when the moment estimates are unreliable.

After warmup, a decay schedule is applied. Cosine decay is the most common choice, smoothly reducing the learning rate to near zero over the remainder of training. Linear decay is also used. Some recipes use a constant learning rate after warmup with a final short decay phase. The choice of schedule affects both the total training cost and the quality of the final model.

The original BERT training recipe established a widely used set of defaults for encoder-style models:

• Learning rate: 1e-4 to 5e-4, with linear warmup over the first 1% of steps, then linear decay to zero
• Weight decay: 0.01
• Beta1: 0.9, Beta2: 0.999
• Gradient clipping: max norm of 1.0
• No weight decay on bias terms or LayerNorm parameters

These values have been validated across many BERT variants and related encoder models. The relatively small weight decay of 0.01 is appropriate for large-scale pretraining, where the dataset is large enough that overfitting is not the primary concern and regularization's main role is to keep weights bounded rather than to prevent memorization.

Decoder-style autoregressive models use somewhat different hyperparameters. The original GPT-3 training recipe, described in the Brown et al. 2020 paper, used:

• Learning rate: 6e-5 for the 175B parameter model, with cosine decay and warmup over 375 million tokens
• Weight decay: 0.1
• Beta1: 0.9, Beta2: 0.95
• Gradient clipping: max norm of 1.0
• No weight decay on bias terms

The choice of Beta2=0.95 rather than 0.999 is notable. A smaller Beta2 gives the second moment estimate a shorter effective memory window, allowing the optimizer to respond more quickly to changes in the gradient distribution as training progresses. For very large models trained on diverse datasets over a long period, the gradient statistics can shift significantly across training phases, and a more responsive second moment estimate can be beneficial. Some subsequent work has found that Beta2 between 0.95 and 0.99 works well for large decoder models, while 0.999 remains fine for smaller models or shorter training runs.

The higher weight decay value of 0.1 in GPT-3 compared to BERT's 0.01 reflects a deliberate choice to apply stronger regularization for large-scale decoder models. At the scale of hundreds of billions of parameters, the risk of the model developing degenerate solutions or excessively large weights in certain components is real, and stronger weight decay provides a useful stabilizing constraint.

When fine-tuning a pretrained model on a smaller downstream dataset, the hyperparameter landscape shifts:

• Learning rate: 1e-5 to 5e-5, often with a short linear warmup over the first few hundred steps
• Weight decay: 0.01 or lower
• Beta1: 0.9, Beta2: 0.999
• Fewer total training steps, shorter decay schedule

Fine-tuning benefits from smaller learning rates because the model starts from a good pretrained initialization and needs precise, controlled adaptation to the target task. Large learning rates during fine-tuning can catastrophically destroy the pretrained representations, causing the model to lose general language understanding while adapting to the specific task. Weight decay is also reduced during fine-tuning: on small datasets, strong regularization can prevent the model from fully adapting to the target task, degrading performance on the very task you are trying to improve.

Because AdamW applies weight decay as $\alpha \lambda \theta$, the learning rate and weight decay are not independent. The effective weight decay per step is $\alpha \lambda$, not just $\lambda$. When you change the learning rate, the effective weight decay changes proportionally. Some practitioners treat the ratio $\lambda / \alpha$ as a stable hyperparameter and rescale weight decay whenever they change the learning rate, though this is a heuristic rather than a rigorously derived rule.

This coupling between learning rate and weight decay also matters when using learning rate schedules. Under a cosine decay schedule, the learning rate decreases to near zero at the end of training, which means the effective weight decay also decreases to near zero. In the final stages of training under cosine decay, the model is receiving almost no regularization. This is sometimes intentional: late-stage training is about fine-tuning the solution rather than constraining it, and reducing weight decay at this stage can allow the model to converge more cleanly to its optimal point without being pulled away by regularization.