Learning Rate Decay › Why the Learning Rate Must Change

Before examining specific schedules, it is worth understanding precisely why a fixed learning rate causes problems at both ends of training. The issue is not merely cosmetic. A fixed learning rate is a fundamental mismatch with the geometry of neural loss surfaces, and understanding that mismatch explains why every major training recipe today uses some form of scheduling.

At the start of training, the model parameters are randomly initialized and far from any good solution. The loss surface at this point is often steep and the gradients are large. A moderate learning rate causes the model to take meaningful steps and make rapid progress. If the learning rate is too small at this stage, training is unnecessarily slow: the model makes tiny moves when large moves are available and beneficial. You are walking when you could be running.

Later in training, the model has moved into a region near a local minimum (or saddle point) where the loss landscape is much flatter. Gradients become smaller and the curvature of the landscape matters more. A learning rate that was appropriate for the steep early landscape now causes the model to bounce back and forth across a shallow basin rather than settling into it. The model never fully converges: it oscillates around the minimum with a loss that fluctuates rather than decreasing smoothly. You are trying to settle into a shallow depression with steps large enough to carry you across it entirely.

A decaying schedule interpolates between these two needs by starting large and shrinking over time. The key design choices are: how quickly to decay, what functional form the decay follows, and what minimum learning rate to allow.

The loss landscape is not uniform throughout parameter space. Near initialization, the landscape is typically dominated by large-scale structure: the model is trying to learn basic patterns (for a language model, that might mean learning that certain words co-occur frequently, or that sentence structure follows certain patterns), and the gradients that point toward these basic patterns are large and relatively consistent across batches. A high learning rate is safe because the direction of improvement is clear.

As training proceeds and the model captures these large-scale patterns, the remaining signal becomes more subtle. The model is now trying to refine fine-grained distinctions, and the gradients that carry this information are smaller. The loss surface near a good local minimum looks like a shallow bowl, and convergence requires increasingly precise steps toward its center. With a fixed step size that is too large, the model overshoots the center every step, landing on the far wall, bouncing back, and oscillating indefinitely.

The mathematical reason for this oscillation comes from the relationship between the learning rate and the curvature. If the maximum eigenvalue of the Hessian (the matrix of second derivatives) at a point is $\lambda_{\max}$, then gradient descent will converge stably only if the learning rate satisfies $\eta < 2 / \lambda_{\max}$. Near a minimum where curvature is high, this constraint is tight. Early in training, where curvature is lower and the landscape is smoother, the same constraint is much looser and larger learning rates are safe.

The practice of reducing the learning rate during training predates deep learning. Stochastic gradient descent theory, developed for convex optimization in the 1950s and 1960s, established convergence guarantees under conditions that require the learning rate to satisfy the Robbins-Monro conditions:

$$\sum_{t=1}^{\infty} \eta_t = \infty \quad \text{and} \quad \sum_{t=1}^{\infty} \eta_t^2 < \infty$$

The first condition ensures the optimizer can reach any part of the parameter space; the second ensures that individual steps eventually become small enough to allow convergence. A constant learning rate satisfies the first condition but not the second (the sum of its squares diverges). The inverse square root schedule, which decays as $t^{-1/2}$, satisfies both conditions, which is one reason it carries such theoretical weight.

Practical deep learning inherited this intuition. Early neural network training papers in the 1980s and 1990s routinely described reducing the learning rate toward the end of training. The specific form varied: some halved it manually, others used exponential decay. The systematic study and naming of these schedules as distinct techniques emerged with the larger-scale experiments of the 2010s, when training runs on ImageNet made hyperparameter sensitivity clearly visible.