Learning Rate Decay › Learning Rate Decay

Training a neural network is not a static process. The learning rate that works well early in training, when the model is far from any good solution and needs large steps to make progress, often becomes a liability later, when the model is close to a good solution and large steps cause it to overshoot. Learning rate decay, also called learning rate scheduling, addresses this mismatch by reducing the learning rate over the course of training.

The intuition is simple: think of gradient descent as hiking down a mountain in heavy fog. Early on, you can stride confidently in roughly the right direction. But as you approach the valley floor, you need to take smaller, more careful steps to avoid stumbling past the lowest point. A fixed learning rate cannot adapt to this changing landscape. A decaying schedule adjusts automatically, enabling fast initial progress and precise final convergence.

This chapter covers the main decay strategies: step decay, exponential decay, inverse square root decay, and cosine annealing. For each, we examine the mathematical formulation, the practical intuition, and when to prefer one over another. We also look at how these schedules interact with adaptive optimizers like Adam, and how modern training pipelines compose multiple schedules together. Along the way, we build concrete intuition for what happens inside optimization when the learning rate changes, and we look at the historical context that shaped each approach.