Learning Rate Decay › Exponential Decay

Exponential decay smooths out the staircase by applying decay at every step rather than at discrete intervals. The learning rate decreases continuously as a smooth exponential curve.

At step $t$, the learning rate is:

$$\eta_t = \eta_0 \cdot e^{-\lambda t}$$

where:

• $\eta_0$: the initial learning rate
• $\lambda$: the decay rate, a positive constant controlling how quickly the rate falls
• $t$: the current training step (or epoch)

An equivalent formulation uses a decay factor $\gamma$ per step:

$$\eta_t = \eta_0 \cdot \gamma^t$$

where $\gamma = e^{-\lambda}$. The two are mathematically identical; the second is often more convenient because you can set $\gamma$ directly (e.g., $\gamma = 0.999$ for slow decay or $\gamma = 0.99$ for faster decay). The relationship $\gamma = e^{-\lambda}$ means $\lambda = -\ln(\gamma)$, so a decay factor of 0.99 corresponds to a rate of approximately 0.01.

The relationship between $\gamma$ and training duration matters enormously. After $T$ steps with per-step factor $\gamma$, the learning rate is $\eta_0 \cdot \gamma^T$. If you want the learning rate to end at, say, 1% of its initial value after 100 epochs, you need $\gamma^{100} = 0.01$, which gives $\gamma = 0.01^{1/100} \approx 0.955$. You can work backward from your desired endpoint to choose $\gamma$.

This backward calculation is a useful habit. Before setting $\gamma$, decide what fraction of the initial learning rate you want at the end of training, and how many steps training will run. The required $\gamma$ follows directly:

$$\gamma = \left(\frac{\eta_{\text{final}}}{\eta_0}\right)^{1/T}$$

This ensures the schedule hits your target endpoint rather than decaying too fast or too slow.

The exponential function has a special property: its rate of change is proportional to its current value. This means exponential decay reduces the learning rate by the same multiplicative factor at every step. Early in training, the absolute reduction is large (because the learning rate is high); later in training, the absolute reduction is small. The relative reduction is constant throughout.

In practice, the effect of this is that exponential decay can shrink the learning rate too quickly, especially with even moderately large $\lambda$. After $t$ steps, the learning rate has shrunk by a factor of $e^{-\lambda t}$, which can become extremely small well before training ends. Practitioners often add a floor: a minimum learning rate below which the schedule does not go, preventing the rate from becoming so small that training effectively stops.

Without a floor, exponential decay with $\gamma = 0.9$ per epoch would reach $0.9^{100} \approx 2.6 \times 10^{-5}$ after 100 epochs starting from $\eta_0 = 0.1$, which is effectively zero. With a floor of $10^{-5}$, the schedule plateaus at that value and training continues to make at least some progress.

The continuous nature of exponential decay is both a strength and a weakness. There are no abrupt transitions to cause loss spikes, but the schedule also never provides the sharp signal of a step drop that can sometimes help the model escape a local plateau. Empirically, exponential decay tends to work well when training is well-behaved and the main concern is smooth, gradual refinement rather than dramatic transitions between phases.