Learning Rate Decay › Summary

Learning rate decay is a fundamental component of neural network training that bridges the gap between fast early learning and precise final convergence. The core problem is that a fixed learning rate cannot simultaneously provide the large steps needed for rapid progress early in training and the small steps needed for stable convergence later. Schedules resolve this by continuously adjusting the learning rate over time.

The key schedules are:

• Step decay: multiplies the learning rate by a factor at fixed epoch intervals, producing a staircase. Easy to interpret and historically common in computer vision. The sharp drops provide clear phase boundaries in training.
• Exponential decay: applies a constant multiplicative factor at every step, producing a smooth exponential curve. Simple but can decay too aggressively without a floor or careful gamma selection.
• Inverse square root decay: falls quickly at first and then levels off, making it well-matched to optimization dynamics. The standard choice for training transformers from scratch, especially combined with linear warmup.
• Cosine annealing: follows a smooth half-cosine curve from maximum to minimum. Widely used for fine-tuning and naturally extended with warm restarts for exploratory training where escaping local minima matters.

All schedules benefit from linear warmup, which prevents instability caused by noisy gradients and uncalibrated optimizer statistics at the start of training. The warmup length is typically 1 to 10 percent of total steps, with longer warmup for larger models and batch sizes.

The choice of schedule interacts with the optimizer. AdamW with cosine annealing and linear warmup has become the default for most NLP fine-tuning work. Inverse square root with warmup remains the standard for large-scale pre-training. SGD with step decay persists in computer vision training recipes that have been validated at scale.

In practice, the peak learning rate is more important to tune than the schedule shape. A well-tuned peak learning rate with any reasonable schedule will outperform a poorly tuned peak with the optimal schedule. Start by finding a good peak learning rate, then refine the schedule if further improvement is needed. The differences between well-tuned schedules are often small compared to the impact of the peak learning rate itself.