Learning Rate Decay › Step Decay

Step decay is the simplest approach and the one historically used in early deep learning. The learning rate is held constant for a fixed number of epochs, then multiplied by a decay factor, then held constant again, and so on. This produces a staircase pattern in the learning rate over time.

The update rule at epoch $t$ is:

$$\eta_t = \eta_0 \cdot \gamma^{\lfloor t / s \rfloor}$$

where:

• $\eta_0$: the initial learning rate
• $\gamma$: the decay factor, typically between 0.1 and 0.5 (so the learning rate is halved, quartered, or reduced by 10x at each step)
• $s$: the step size in epochs (how many epochs between drops)
• $\lfloor t / s \rfloor$: the floor of $t / s$, which counts how many complete steps have elapsed

For example, with $\eta_0 = 0.1$, $\gamma = 0.1$, and $s = 30$, the learning rate starts at 0.1, drops to 0.01 at epoch 30, drops to 0.001 at epoch 60, and so on. Each drop reduces the learning rate by exactly a factor of ten.

The floor function $\lfloor t / s \rfloor$ is what creates the staircase: it stays constant for $s$ consecutive epochs, then increments by exactly one. This means the learning rate at epoch 29 is identical to the learning rate at epoch 0, and the learning rate at epoch 30 is exactly $\gamma$ times that.

Step decay appeals to practitioners because it is easy to understand and diagnose. You can look at a training loss curve and see exactly when each drop happened. If the loss plateaus before a drop, that suggests the drop might be too late. If the loss jumps up briefly after a drop, the new learning rate might be too small. This interpretability makes step decay a useful debugging tool. Every phase of training is clearly delineated by the drop, and you can reason about what the model was doing in each phase.

The staircase structure also means training can be naturally organized into phases. Each phase trains at a fixed learning rate until the loss stops improving, then the next phase begins at a lower rate to refine the solution. Many early ResNet and VGG training recipes used this pattern: train for 30 epochs at 0.1, drop to 0.01, train 30 more, drop to 0.001, and finish. He et al.'s original ResNet paper for ImageNet used exactly this recipe, and the robustness of this approach across many architectures is what established its dominance in computer vision through the early 2010s.

The main limitation is rigidity. The schedule is specified in advance and does not adapt to the observed training progress. If a particular run converges faster or slower than expected, the drops happen at the wrong time. A model that has already plateaued two epochs before the scheduled drop is waiting unnecessarily. A model that is still making rapid progress when the drop arrives gets its learning rate cut during a productive phase. Some practitioners address this by monitoring validation loss and dropping the rate when it plateaus rather than at fixed epochs, a variant called ReduceLROnPlateau. This adaptive version maintains the staircase structure while letting training dynamics determine when each step occurs.

The choice of $\gamma$ has a significant impact on the training dynamics. A very aggressive factor like $\gamma = 0.1$ (ten-fold reduction) means that after the drop, the model is taking steps one-tenth as large. This can effectively freeze learning on some parameters: if the gradient for a particular parameter is small at the pre-drop learning rate, it becomes negligible after a ten-fold reduction. The drop essentially shifts the focus of training to parameters with larger gradients.

A gentler factor like $\gamma = 0.5$ (halving) is less dramatic. The model slows down but does not stop making meaningful progress on any parameter. Multiple gentler drops can sometimes achieve the same end result as one aggressive drop while maintaining smoother loss curves.

The step size $s$ determines how long the model trains at each level. Setting $s$ too small means the model does not have time to exploit each learning rate before it drops. Setting $s$ too large means the model may have plateaued long before the next drop. The optimal $s$ is essentially the number of epochs it takes for the model to plateau at each learning rate level, which depends on the dataset, architecture, and optimizer.