Learning Rate Decay › Schedules and Adaptive Optimizers

Learning rate decay was designed primarily for SGD, where the learning rate directly controls step size. Adaptive optimizers like Adam complicate this picture considerably, and understanding the interaction is important for using schedules correctly.

Adam maintains a per-parameter estimate of the learning rate through its second moment estimate $v_t$. The effective step size for parameter $\theta_i$ is approximately $\eta / \sqrt{v_t + \epsilon}$, where $v_t$ tracks the historical squared gradients. Parameters with large, consistent gradients get small effective steps, and parameters with small or variable gradients get larger steps.

The global learning rate $\eta$ in Adam functions as an overall scaling factor on top of this per-parameter adaptation. When you apply a learning rate schedule on top of Adam, you are scaling this already-adapted step size. The schedule affects all parameters equally by a common multiplier, while Adam's internal adaptation continues to differentiate between them.

This combination generally works well in practice. The schedule handles the global progression from fast to slow learning, while Adam handles the per-parameter optimization. Most practitioners apply standard decay schedules to Adam without modification and find that this works reliably.

One might wonder whether schedules are necessary for Adam at all, given that Adam already adapts. The answer is yes, for several reasons.

First, Adam's adaptive mechanism operates on the ratio of first to second moments, not on the absolute magnitude of the learning rate. The global $\eta$ still controls the overall scale of updates, and a well-calibrated schedule ensures this scale shrinks appropriately as the model approaches convergence.

Second, Adam can get stuck oscillating around a minimum with a fixed learning rate for the same geometric reason as SGD. The curvature of the loss surface near the minimum sets a stability limit on the step size, and Adam's internal adaptation does not automatically reduce steps below this limit. A decaying schedule that brings $\eta$ below the stability threshold is what allows the model to settle.

Third, for fine-tuning, the schedule interacts with the model's pre-trained weights in an important way. Starting with a large Adam learning rate for fine-tuning can corrupt pre-trained representations, even with Adam's adaptation. A careful warmup followed by a decaying schedule protects the pre-trained weights by keeping updates small early on.

One subtlety arises with weight decay. Standard Adam with L2 regularization applies a decay term that is scaled by the adaptive learning rate, meaning parameters with large gradients receive less weight decay. This is mathematically inconsistent: weight decay is supposed to pull all parameters toward zero uniformly, but coupling it to the adaptive learning rate means it is applied differently to different parameters.

AdamW, introduced by Loshchilov and Hutter in 2019, fixes this by decoupling the weight decay from the gradient update: weight decay is applied at a fixed rate regardless of the adaptive step size. When using AdamW, the learning rate schedule affects gradient-based updates, and weight decay is controlled separately.

This decoupling matters for schedule design because it means the learning rate schedule in AdamW controls only the gradient step, not the regularization strength. You can set the learning rate schedule aggressively without worrying about undermining the weight decay regularization. The two can be tuned independently, which simplifies hyperparameter search.