Cosine Learning Rate Schedule › Historical Context and Origin

The cosine schedule did not appear from nowhere. It has a clear intellectual lineage rooted in optimization theory and the physics of annealing.

The broader concept of annealing in optimization descends from the metallurgical process of slowly cooling a material to reduce crystalline defects. In physical annealing, a metal is heated until atoms can move freely, then cooled gradually so they have time to settle into a low-energy, ordered arrangement. Rapid cooling traps defects; gradual cooling allows the system to explore its energy landscape and find a more globally stable configuration. Simulated annealing, introduced by Kirkpatrick, Gelatt, and Vecchi in 1983, imported this intuition into numerical optimization: explore the solution space while "temperature" is high, then exploit as temperature cools. The learning rate in gradient descent plays the role of temperature, and its gradual reduction mirrors the cooling schedule of simulated annealing.

Early neural network training used fixed learning rates or manually designed step-drop schedules because practitioners did not yet have the compute resources to run enough experiments to compare schedule shapes systematically. The AlexNet paper from 2012 used a step decay schedule that reduced the learning rate by a factor of 10 whenever the validation error stopped improving, with three manual reductions over the course of training. This "reduce on plateau" approach worked well for relatively short training runs where a practitioner could monitor validation curves and decide when to drop the rate, but it did not scale to the longer, unsupervised pretraining runs that would come later.

The cosine annealing schedule was formalized in a neural network context by Loshchilov and Hutter in their 2016 ICLR paper "SGDR: Stochastic Gradient Descent with Warm Restarts." That paper's primary contribution was the warm restart mechanism, but it also established the cosine annealing formula itself as a principled choice for the decay within each cycle. The smooth interpolation using the cosine function offered a natural improvement over step decay without the manual threshold-setting. The paper demonstrated that cosine annealing with restarts improved performance on CIFAR-10 and CIFAR-100 relative to both step schedules and simple cosine without restarts.

Large language model training adopted the cosine schedule without restarts, dropping the restart component but retaining the smooth cosine decay. GPT-2 (2019) used a warmup-plus-cosine design. GPT-3 (2020) formalized it with explicit parameters. By the time LLaMA appeared in 2023, the warmup-cosine recipe was so standard that its use needed almost no justification in the paper. What began as an ablation result from an optimization paper became the de facto default for training language models at scale.

Understanding this history matters because it reveals that the cosine schedule's dominance is partly empirical and partly path-dependent. It has been tested and validated at every scale from millions to hundreds of billions of parameters. Its smooth shape avoids the worst failure modes of step decay. Its simplicity makes it easy to reason about and reproduce. None of this means that alternative schedules could not perform as well or better, and active research continues to explore this space. But it does mean that when you train a language model with a cosine schedule, you are following a well-trodden path with a large body of existing knowledge to guide your parameter choices.