Learning Rate Warmup › Summary

Learning rate warmup starts training with a very small learning rate and gradually increases it to the target value over a defined number of steps. The core motivation is threefold: it gives Adam's second-moment estimates time to stabilize, it allows gradients to find more reliable directions, and it prevents transformer attention weights from saturating before meaningful representations form.

Linear warmup is the dominant strategy, computing $\eta_t = \eta_{\max} \cdot (t / T_{\text{warmup}})$ during the warmup phase before transitioning to a decay schedule. The warmup duration is typically 0.5-2% of total training steps, with a theoretical minimum of approximately $1 / (1 - \beta_2)$ steps based on Adam's statistics stabilization. For the standard $\beta_2 = 0.999$, this means at least 1,000 warmup steps.

When scaling to larger batch sizes, both the learning rate and warmup duration should scale proportionally with the batch size ratio. This linear scaling rule is essential for preserving training stability when increasing parallelism. The square-root scaling rule provides a more conservative alternative for very large batch sizes where gradient variance reduction plateaus.

Beyond linear warmup, variants like polynomial warmup, cosine warmup, and the Warmup-Stable-Decay schedule offer different tradeoffs. For most applications, linear warmup is the right default choice. When warmup fails, characteristic loss spikes and gradient norm patterns allow you to diagnose the problem and adjust either the warmup duration or the peak learning rate.

The next chapter covers Learning Rate Decay, which describes what happens after warmup ends: how cosine schedules, step decay, and inverse square-root decay shape the learning rate throughout the bulk of training.