Cosine Learning Rate Schedule › Limitations and Practical Considerations

The cosine schedule is not a silver bullet, and understanding its failure modes helps you adapt it appropriately.

One important limitation is sensitivity to the total training budget. The cosine curve is defined relative to the total number of steps $T$, which means you must fix your training duration before you begin. If you need to stop training early or extend it, the schedule behaves unexpectedly: stopping early leaves the learning rate higher than intended at the end, while extending training pins the rate at $\eta_{\min}$ for the extra steps. This contrasts with some more adaptive schedules that respond to validation metrics rather than step counts. In practice, most large-scale training runs fix the compute budget ahead of time, so this limitation is manageable, but it requires forethought.

Another subtlety is that the cosine schedule decays most rapidly in the middle of training. If your model has not found a reasonable region of parameter space by the midpoint, the accelerating decay phase can be disruptive rather than helpful. This is why the warmup phase is so commonly paired with cosine decay: warmup ensures the model has made initial progress before the cosine curve begins decelerating. Models trained without warmup into a cosine schedule sometimes exhibit a "plateau then collapse" pattern, where the flat early-training behavior is followed by a sudden rapid descent as the cosine derivative peaks.

Warm restarts add a new dimension of complexity. Each restart risks disturbing a converging model and requires the optimizer to re-explore the loss landscape. If cycle lengths are too short, the model never fully converges within a cycle before being disturbed again. If they are too long, restarts offer little benefit over a single cosine decay. Tuning $T_0$ and $T_{\text{mult}}$ is often done empirically, which adds experimental overhead. For most language model training runs, practitioners have moved toward single-cycle cosine decay without restarts, reserving restarts for cases where snapshot ensembling is a goal.

The combination of warmup and cosine decay introduces two more hyperparameters ($T_{\text{warmup}}$ and $\eta_{\min}$) on top of the base learning rate, making the full schedule four-dimensional. Each parameter interacts with the others. A very long warmup followed by aggressive cosine decay can starve the model of a useful exploratory phase, while a very short warmup into a gentle decay may fail to stabilize early training. Standard guidance (1-4% of steps for warmup, $\eta_{\min} = \eta_{\max} / 10$) provides good starting points, but some tuning is usually required for novel architectures.

The trapezoidal schedule, discussed earlier, addresses the budget-sensitivity problem by decoupling the decay from the total training duration. However, it introduces its own sensitivity: the quality of training depends heavily on the cooldown duration and shape. Too short a cooldown and the model does not fully converge; too gradual and the plateau phase ends earlier than necessary. Research comparing trapezoidal and cosine schedules at large scale is still ongoing, and the field has not yet converged on a clear winner.

A number of common training pathologies can be traced to schedule misconfiguration, and recognizing these patterns early can save significant compute.

Loss spikes at training start almost always indicate that the peak learning rate is too high, or that the warmup is too short. The optimizer takes massive steps on the random initialization before establishing a stable gradient direction. Reducing $\eta_{\max}$ by a factor of 3-5 or doubling the warmup duration usually resolves this. A useful diagnostic: if loss spikes resolve within 50-100 steps and the loss curve then continues normally, the spike was a minor warmup artifact. If loss continues oscillating or diverges, the peak learning rate is the likely culprit.

Stalling in mid-training (loss plateauing well above the expected final value, for many steps) can indicate that the cosine decay has progressed too quickly to a regime where updates are too small to escape the current region. This often happens when $\eta_{\max}$ was set too conservatively. One remedy is to restart training with a higher peak or a longer warmup that allows more time at the peak learning rate. Another sign of this failure: the cosine curve is already 60-70% decayed but the loss has not yet entered its expected rapid descent phase.

Loss oscillation in the final phase (the last 10-20% of training) can indicate that $\eta_{\min}$ is too large. If the minimum learning rate is 30% or 50% of the peak, the optimizer is still taking substantial steps when it should be settling. Reducing $\eta_{\min}$ to 5-10% of the peak typically smooths the final convergence.

Unexpected loss increase after extending training is a sign that the cosine schedule has already decayed to $\eta_{\min}$ and the extra steps are essentially running with the optimizer frozen at the minimum rate. In some optimizers and loss surfaces, running at a near-zero learning rate for many steps can cause the model to drift slightly as momentum terms interact with nearly-zero updates in unexpected ways. The safest approach when extending training is to either restart the schedule from the current learning rate (effectively treating the extension as a new cosine cycle starting from wherever the schedule left off) or switch to the trapezoidal approach from the beginning.

The cosine schedule interacts with batch size in a way that is easy to overlook. As covered in the context of learning rate warmup, the linear scaling rule suggests that if you multiply your batch size by $k$, you should multiply your peak learning rate by $k$ as well to maintain the same effective optimization trajectory. This rule applies directly to the cosine schedule: both $\eta_{\max}$ and $\eta_{\min}$ should scale with batch size, keeping their ratio constant.

However, the warmup duration also needs to adjust when batch size changes. With a larger batch, each step covers more data, and the variance of gradient estimates is lower. The optimizer orients itself more quickly, meaning it needs fewer warmup steps in terms of iterations (though about the same number of data samples). A practical rule of thumb is to scale warmup steps inversely with batch size: if you double the batch size, halve the number of warmup steps, so the total number of samples seen during warmup remains constant.

The total number of decay steps $T$ also interacts with batch size through the total token count. If your budget is fixed in tokens rather than in steps, doubling the batch size halves the number of steps for the same token count. The cosine schedule's $T$ should be set to the token-equivalent step count, not an arbitrary step count, to ensure the decay profile is consistent regardless of batch size.

Despite these considerations, the cosine schedule performs reliably across a wide range of scales and architectures. The GPT series from OpenAI, the LLaMA family from Meta, Mistral, and countless smaller research models all report using cosine decay as their default schedule. Its widespread adoption has produced substantial empirical evidence about good default values, making it far easier to configure than more complex adaptive schedules that change behavior based on validation metrics.

Looking ahead, the next chapter on large batch training discusses how schedule parameters interact with batch size, and why the linear scaling rule for learning rates must be applied carefully when the cosine schedule is in use.