Learning Rate Decay › Decay Scheduling in Practice

The main schedules covered above can be augmented with additional techniques for practical training pipelines. This section covers the remaining standard variants, discusses how to compose multiple schedules, and addresses practical choices that come up repeatedly.

Linear decay reduces the learning rate in equal steps across training:

$$\eta_t = \eta_0 \cdot \left(1 - \frac{t}{T}\right)$$

where $T$ is the total number of training steps. Linear decay is easy to understand and widely used in language model fine-tuning. Many Hugging Face training scripts use linear warmup followed by linear decay as their default schedule.

The key property of linear decay is predictability: the learning rate at any step is a straightforward linear interpolation between the starting and ending values. This makes it easy to reason about training progress. If you are at 50% of total steps, the learning rate is exactly 50% of the initial value. If you extend or shorten training, you can immediately compute the new schedule without any complex parameter relationships.

Linear decay is also well-matched to fine-tuning pre-trained models. In fine-tuning, the goal is to adapt the model to a new task while preserving the general knowledge from pre-training. The learning rate needs to start small enough to not disrupt pre-trained weights too violently, and then decrease further to ensure precise convergence. A linear schedule that starts at a modest learning rate like $2 \times 10^{-5}$ and decreases to zero over, say, three epochs is exactly this kind of gentle, predictable refinement.

Polynomial decay generalizes both linear and inverse square root decay:

$$\eta_t = \left(\eta_0 - \eta_{\text{end}}\right) \cdot \left(1 - \frac{t}{T}\right)^p + \eta_{\text{end}}$$

where $p$ is the polynomial power and $\eta_{\text{end}}$ is the final learning rate. Setting $p = 1$ gives linear decay. Setting $p = 0.5$ gives a schedule that drops quickly initially and then slows, resembling inverse square root. Setting $p = 2$ gives a quadratic curve that drops slowly at first and quickly near the end, the opposite behavior.

The power $p$ is rarely tuned in practice because linear and cosine are simpler and well-understood. Polynomial decay appears in some large-scale training recipes where precise control over the decay shape is desired. The key insight is that the exponent $p$ controls the skew of the decay: smaller $p$ means more reduction happens early, larger $p$ means more reduction happens late.

Every schedule should have a non-zero minimum learning rate. Without a floor, exponential or inverse square root schedules approach zero asymptotically. At very small learning rates, training effectively stops: gradients are multiplied by a negligible factor and weights barely move. A floor of $10^{-5}$ or $10^{-6}$ ensures that training continues to make at least some progress.

The floor is especially important for schedules used in fine-tuning over many epochs. If you accidentally set $\gamma$ too aggressively for an exponential schedule, the learning rate could reach the floor very early in training, wasting compute on steps where the parameters barely change. Monitor the learning rate reached during training as well as the schedule configuration.

Most modern training pipelines compose a warmup phase with one of the main decay schedules. The warmup brings the learning rate from zero to its peak value over the first few hundred or thousand steps, and then one of the decay schedules takes it from the peak down to the floor.

The transition from warmup to decay is handled differently depending on the framework. In some implementations, warmup and decay are separate schedule objects that are chained together. In PyTorch's get_cosine_schedule_with_warmup from the Hugging Face transformers library, the two phases are unified into a single function that changes behavior at the warmup boundary.