Cosine Learning Rate Schedule › Summary

The cosine learning rate schedule applies the cosine function to interpolate the learning rate smoothly from a peak value $\eta_{\max}$ down to a minimum $\eta_{\min}$ over a fixed number of training steps. Its key properties are:

• The decay rate is slow at the start and end of training and fastest in the middle, matching the natural phases of gradient descent.
• Cosine decay consistently outperforms step decay (no abrupt drops) and slightly outperforms linear decay in final model quality, because it front-loads exploration and back-loads convergence.
• The derivative of the cosine schedule is a scaled sine function, which is zero at the boundaries and maximal at the midpoint. This gives the schedule its characteristic pacing.
• Warm restarts (SGDR) periodically reset the learning rate, helping the optimizer escape sharp minima and enabling snapshot ensembling. They are most useful in computer vision and fine-tuning scenarios, and are less common in large-scale LLM pretraining.
• The most common production configuration combines a short linear warmup (1-5% of total steps) with a long single-cycle cosine decay.
• Critical hyperparameters are $\eta_{\max}$, $\eta_{\min}$ (typically $\eta_{\max}/10$), total decay steps $T$, and (when using restarts) cycle length $T_0$ and multiplier $T_{\text{mult}}$.
• The trapezoidal schedule tolerates variable training durations better, at the cost of a concentrated cooldown phase.

The cosine schedule is now the de facto standard for training large language models, not because it is theoretically optimal but because it is smooth, interpretable, and empirically reliable across a broad range of training scenarios. The worked numerical examples above make clear exactly how the formula allocates the learning rate budget across training, and the PyTorch implementations show how to apply these principles directly in practice.