Cosine Learning Rate Schedule › Trapezoidal Schedules: A Modern Alternative

A more recent alternative that has gained attention for large-scale language model training is the trapezoidal (or "trapezoid") schedule. Rather than decaying from the very beginning, the trapezoidal schedule keeps the learning rate at its peak for most of training, then drops it sharply in a brief cooldown phase at the end.

The schedule has three phases:

1. Warmup: Linear ramp from 0 to $\eta_{\max}$ over $T_{\text{warmup}}$ steps.
2. Stable: Constant learning rate at $\eta_{\max}$ for $T_{\text{stable}}$ steps (typically 80-90% of total training).
3. Cooldown: Cosine or linear decay from $\eta_{\max}$ to $\eta_{\min}$ over $T_{\text{cooldown}}$ steps.

The motivation for this schedule is compute-budget flexibility. With a cosine schedule, you must decide the total training duration $T$ in advance, because the decay rate depends on $T$. If you want to train for 20% longer (perhaps because the model is still improving), the entire cosine curve needs to be rescaled. With a trapezoidal schedule, you can extend the stable phase arbitrarily without affecting the decay curve: just train for longer at the constant peak rate, then apply the same cooldown at the end.

Recent work including the Chinchilla analysis and follow-up scaling papers has noted that the final cooldown phase is disproportionately important for model quality. A model trained at constant learning rate for a very long time, then cooled down over a few thousand steps, can match or exceed the quality of a model trained with a cosine schedule for the same total compute. This property makes the trapezoidal schedule particularly useful for continual or incremental training, where the exact endpoint is not known in advance.