Cosine Learning Rate Schedule › Choosing the Right Schedule for Your Task

With the core schedules covered, it helps to have a practical framework for deciding which one to apply to a given training scenario. The choice depends on your compute budget, whether the training duration is fixed, whether you need multiple model checkpoints, and how much hyperparameter tuning you can afford.

If you are training a model from scratch with a fixed compute budget and you know the total number of steps in advance, the warmup-plus-cosine schedule is the safest default. It matches the training recipes of most publicly released LLMs, which means you can borrow hyperparameters from similar models as starting points. The warmup duration should cover 1-4% of total steps (2000 warmup steps out of 100,000 total is typical), and the minimum learning rate should be set to 10% of the peak. Starting from a peak learning rate that worked for a similar-sized model and adjusting up or down based on early loss curves is a reliable strategy.

The main risk with cosine decay in fixed-budget pretraining is choosing the wrong peak learning rate. If the peak is too high, training instabilities or loss spikes will appear early. If it is too low, the model will converge more slowly than necessary, wasting compute. Running a short learning rate range test (training for a few thousand steps at several candidate peak rates and comparing early loss trajectories) before committing to a full run saves significant compute at scale.

If you are not certain how long training will run, perhaps because you are monitoring a validation metric and might stop early or extend the run depending on what you observe, the trapezoidal schedule handles this far more gracefully than cosine decay. You commit only to the cooldown duration and shape, and the stable phase can be extended arbitrarily. The cosine schedule's decay is defined relative to the total budget $T$, so changing $T$ mid-run requires either rescaling the decay or accepting that the learning rate at the current step no longer matches the intended schedule.

For continual learning scenarios, where a model is periodically updated on new data without restarting from scratch, the trapezoidal schedule is also preferred. Each update cycle consists of a brief warmup from the current learning rate (not from zero), a stable phase on the new data, and a cooldown. The cosine schedule does not naturally accommodate this pattern because its decay rate is coupled to the total training horizon.

Fine-tuning a pretrained model introduces different constraints from pretraining. The model is already near a good solution in its parameter space, and the risk of catastrophic forgetting (discussed in detail in the Finetuning Fundamentals chapter) means you want smaller, more careful updates throughout training. The peak learning rate should be roughly 10-100 times smaller than what you would use for pretraining, with typical values in the range of $1 \times 10^{-5}$ to $1 \times 10^{-4}$.

With these smaller learning rates, warm restarts can be useful because the risk of a restart disturbing the pretrained representations is lower when the restart amplitude is small. A restart from $1 \times 10^{-5}$ is far less disruptive than a restart from $3 \times 10^{-4}$. If you are fine-tuning on a small dataset where the model can overfit quickly, short restart cycles help the optimizer escape any sharp minima that form early in training.

The warmup phase is typically shorter in fine-tuning, often just 50-100 steps, because the pretrained model's parameters are already well-initialized and the optimizer does not need as much time to establish stable gradient directions.

If you have limited computational resources for hyperparameter tuning, the cosine schedule's main advantage is that its shape is largely fixed once you specify $\eta_{\max}$, $\eta_{\min}$, and $T$. Relative to an adaptive schedule like ReduceLROnPlateau, which requires specifying a validation metric, a patience threshold, a reduction factor, and a minimum learning rate, the cosine schedule is simpler to configure and more predictable.

For practitioners who can afford systematic hyperparameter sweeps, the most impactful parameters to tune in a cosine schedule are, roughly in order of importance: the peak learning rate, the warmup duration, and the minimum learning rate. The schedule shape itself (cosine vs. linear) is typically not the most sensitive hyperparameter; the absolute scale of the learning rate matters far more. This means you can fix the cosine schedule structure and focus your tuning budget on the peak value and warmup, which is an efficient allocation of experimental resources.