Cosine Learning Rate Schedule › How Real Models Use Cosine Schedules

The cosine schedule has been adopted by nearly every large language model developed in the past several years. Examining the published training recipes of prominent models reveals both the universality of the cosine approach and the specific parameter choices that practitioners have converged on.

GPT-3 (Brown et al., 2020) uses cosine decay to reduce the learning rate from $\eta_{\max}$ to $10\%$ of its peak value over the full training run. The warmup lasts for the first 375 million training tokens, which represents roughly 0.05% of the 300 billion token training corpus. The model uses a peak learning rate of $6 \times 10^{-5}$ for the 175-billion-parameter version. GPT-3 trained smaller and larger models with correspondingly adjusted peak learning rates, following a rough inverse square-root scaling with model size: larger models use lower peak learning rates, showing the intuition that wider parameter spaces benefit from more conservative steps.

LLaMA (Touvron et al., 2023) follows a nearly identical pattern: cosine decay with $\eta_{\min} = \eta_{\max} / 10$, a warmup over the first 2000 steps, and a peak learning rate of $3 \times 10^{-4}$ for the 7-billion-parameter model. The 65-billion-parameter model uses a lower peak of $1.5 \times 10^{-4}$. The LLaMA paper provides some of the clearest documentation of cosine schedule usage among published large models, explicitly noting that the minimum learning rate rule of $\eta_{\min} = \eta_{\max} / 10$ was adopted from earlier empirical guidance.

LLaMA 2 (Touvron et al., 2023) extended the original LLaMA recipe with longer training horizons (up to 2 trillion tokens for the 7B model) while retaining the same schedule structure. One practical detail from LLaMA 2's training: the model's context length was increased midway through training, and the schedule was adjusted accordingly. The cosine decay was restarted from its current value at the point of context length expansion, treating the extended training as a continuation of the original schedule. This illustrates an important practical consideration: even with a planned cosine decay, real training runs often require ad-hoc adjustments that the rigid schedule does not account for.

Mistral-7B uses cosine decay with a warmup of 2000 steps and a peak learning rate of $3 \times 10^{-4}$, trained on 1 trillion tokens. Like LLaMA, the minimum learning rate is 10% of the maximum. Mistral's architecture differs from LLaMA in using sliding window attention and grouped query attention, but the training schedule is nearly identical, suggesting the cosine recipe generalizes across these architectural variations.

PaLM (Chowdhery et al., 2022) uses cosine decay from $1 \times 10^{-2}$ down to $1 \times 10^{-4}$ for the 540-billion-parameter model, with linear warmup over 10,000 steps. The higher initial learning rate reflects PaLM's use of the Adafactor optimizer rather than Adam. Adafactor normalizes gradient magnitudes differently from Adam, allowing it to work well with higher raw learning rates. The cosine schedule structure is identical; only the scale differs.

Gemma (Team et al., 2024) from Google DeepMind also adopts the cosine schedule with warmup, using a peak learning rate of $1 \times 10^{-3}$ and a cosine decay to $1 \times 10^{-4}$ over the course of training. The relatively high peak learning rate, higher than LLaMA's, reflects the use of smaller batch sizes and different gradient accumulation strategies, again illustrating how the cosine schedule's structure transfers across configurations while the specific values need tuning.

Falcon (Penedo et al., 2023) trains with a cosine schedule from $3 \times 10^{-4}$ down to $3 \times 10^{-5}$, trained on up to 1 trillion tokens. Falcon's training run is notable for its use of large batch sizes (up to 2 million tokens per batch), which required careful adjustment of the peak learning rate using linear scaling. Even with this scaling, the cosine decay structure remained unchanged.

The consistency across these very different models (in scale, architecture, training data, and optimizer variant) suggests that the cosine schedule tolerates these variations. The shared pattern is: a short warmup (a few hundred to a few thousand steps), a cosine decay over the full training run, and a minimum learning rate at roughly 10% of the peak. This recipe transfers well across scales and datasets.

One pattern that emerges from these model comparisons is a rough relationship between model size and peak learning rate. Larger models tend to use lower peak learning rates, and this is not purely empirical. The Chinchilla scaling paper (Hoffmann et al., 2022) identified that the optimal number of training tokens scales linearly with model parameters. For a given compute budget, larger models are trained on fewer tokens (though still many), and their parameter updates span a much larger space. These two factors together suggest that larger models benefit from more conservative per-step updates.

A commonly cited approximation is that peak learning rate scales with model size $N$ roughly as $\eta_{\max} \propto N^{-0.5}$, though empirical fits vary across different research groups and architectures. In practice, the exact scaling exponent matters less than the general direction: if you double your model size, experimenting with a learning rate of $\eta_{\max} / \sqrt{2}$ rather than $\eta_{\max}$ is a reasonable first step. The cosine schedule's decay structure remains unchanged; you are adjusting only the peak value.

This relationship between model size and learning rate also interacts with the choice of $\eta_{\min}$. If $\eta_{\min}$ is defined as a fixed fraction of $\eta_{\max}$, as in the LLaMA convention of $\eta_{\min} = \eta_{\max} / 10$, then reducing $\eta_{\max}$ also reduces $\eta_{\min}$ proportionally. This is generally the right behavior: finer final updates correspond to a finer resolution of parameter space, which is appropriate for larger, more expressive models settling into more precise local minima.