Cosine Learning Rate Schedule › Cosine Schedule Parameters

A cosine schedule has four tunable parameters. Understanding what each controls helps you set them deliberately rather than by guessing.

The maximum learning rate $\eta_{\max}$ determines the initial optimization speed. In a warmup-plus-cosine schedule (by far the most common configuration in large model training), $\eta_{\max}$ is the peak the warmup ramps toward. Setting this value follows the same guidelines as setting a fixed learning rate: too large and gradients explode, too small and training stalls. Linear scaling with batch size (multiply by a factor of $B / B_{\text{ref}}$ when increasing batch size from $B_{\text{ref}}$ to $B$) is a common heuristic, though the right value for a given model ultimately requires some experimentation.

For AdamW-based LLM training, values in the range $[1 \times 10^{-4}, 3 \times 10^{-4}]$ have been widely used at scales from a few hundred million parameters up to the tens of billions. Smaller learning rates around $1 \times 10^{-5}$ are typical for fine-tuning, where the model is already near a good solution and large steps risk destroying the pretrained representations.

The minimum learning rate $\eta_{\min}$ is the floor the cosine curve decays toward. In many implementations this is simply 0, meaning the learning rate completely vanishes at the end of training. However, setting $\eta_{\min}$ to a small positive value like $10^{-6}$ or $\eta_{\max} / 100$ can improve final performance by allowing the optimizer to keep making small corrections through the last steps. In LLaMA, for example, $\eta_{\min}$ is set to $\eta_{\max} / 10$, meaning the final learning rate is 10% of the peak.

The choice of $\eta_{\min}$ has a subtle but real effect on the final model quality. Setting it to exactly zero creates a situation where the optimizer effectively stops learning in the final steps, which can be desirable if you want the model to fully converge to a minimum. Setting it to a non-zero value keeps a residual update signal alive, which helps when the training data has been partially exhausted and the model is in danger of overfitting. Many practitioners use $\eta_{\min} = \eta_{\max} / 10$ as a safe default that avoids both extremes.

The parameter $T$ represents the total number of steps over which the cosine decay runs. In a single-cycle schedule this equals the full training duration. In a warmup-plus-cosine schedule, $T$ typically spans from the end of warmup to the end of training, though it can extend slightly beyond the actual training duration as a practical trick. If $T$ is set smaller than the actual training length, the learning rate will linger at $\eta_{\min}$ for the remaining steps, which is effectively the same as turning off the optimizer. Setting $T$ slightly larger than training means the schedule has not fully decayed at the end, keeping the learning rate from reaching zero.

One subtle design choice is whether to count warmup steps as part of $T$. In some frameworks, $T$ refers to the total training steps including warmup, and the cosine decay starts from zero alongside the warmup. In others, warmup is treated as a separate phase and cosine decay begins after warmup ends, with $T$ counting only the post-warmup steps. The difference matters: if you set $T = 10000$ but warmup takes 1000 steps, you want to be explicit about whether the cosine curve should span 10000 or 9000 steps. Many practitioners prefer setting $T$ equal to the total training budget and letting the warmup logic handle the first phase separately, which is the pattern we follow in our implementation below.

When using warm restarts, $T_0$ (the first cycle length) and $T_{\text{mult}}$ together determine the restart pattern. Short cycles with $T_{\text{mult}} = 1$ create frequent restarts that help exploration but may prevent the model from converging deeply into any minimum. Long initial cycles with $T_{\text{mult}} = 2$ start with a substantial refinement phase and progressively lengthen subsequent cycles, which often performs better for large models. A common starting point for language model training is to skip restarts entirely and use a single long cosine cycle, reserving restarts for computer vision tasks where snapshot ensembling benefits are more established.

When you do use restarts, the cycle length $T_0$ should be at least long enough for the model to make meaningful progress within each cycle. For typical LLM fine-tuning runs of 5,000-20,000 steps, $T_0$ values between 500 and 2000 are common. For pretraining runs of hundreds of thousands of steps, $T_0$ values of 10,000-50,000 steps are more appropriate.