Learning Rate Warmup › Warmup Duration

How long should warmup last? This is one of the most commonly asked hyperparameter questions in large-scale training, and the answer depends on several factors.

For most transformer training runs, warmup should last between 0.5% and 2% of total training steps. Common specific recommendations from landmark training runs:

• BERT used 10,000 warmup steps out of approximately 1,000,000 total steps (1%)
• GPT-2 used 2,000 warmup steps
• T5 used 10,000 warmup steps
• LLaMA and its successors typically use 2,000 warmup steps

For smaller models or shorter training runs, you can use fewer warmup steps. A model trained for 10,000 steps might use only 100-500 warmup steps. For very large models (billions of parameters) trained for trillions of tokens, warmup up to 2,000-5,000 steps is typical because the peak learning rate is often higher and the consequences of early instability are more severe.

One important observation is that the absolute number of warmup steps matters more than the percentage. A model trained for 200,000 steps and one trained for 50,000 steps may both use 2,000 warmup steps, because the constraint on warmup duration comes from the optimizer's statistics stabilization time, not from the total training budget.

The minimum warmup duration is bounded by how long it takes for Adam's second-moment estimate $v_t$ to become reliable. For a parameter with gradient variance $\sigma^2$, the bias-corrected second-moment estimate after $t$ steps has variance:

$$\text{Var}(\hat{v}_t) \approx \frac{2\sigma^4 \beta_2^t}{t(1-\beta_2)^2}$$

where:

• $\hat{v}_t$ is Adam's bias-corrected second-moment estimate at step $t$
• $\sigma^2$ is the true variance of the gradient for a given parameter
• $\beta_2$ is Adam's second-moment decay rate (typically 0.999)

This estimate becomes reliable, meaning its coefficient of variation drops below 10%, approximately when the number of steps satisfies:

$$t > \frac{1}{1 - \beta_2}$$

With $\beta_2 = 0.999$, that threshold is $t > 1 / (1 - 0.999) = 1000$ steps. This matches the empirical observation that 1,000 to 2,000 warmup steps work well for most runs. The formula tells us that the warmup duration is essentially the "memory span" of Adam's second moment estimator: you need to collect enough samples for the exponential moving average to represent the recent gradient distribution accurately.

There is a direct connection to the $\beta_2$ parameter: using a smaller $\beta_2$ (such as 0.99 or 0.95) allows Adam's second-moment estimate to adapt faster, because the exponential moving average forgets old samples more quickly and new samples have more weight. This means you can use shorter warmup with a smaller $\beta_2$. Some recent work on adaptive optimizers explicitly reduces $\beta_2$ for the first few thousand steps for this reason. But for the standard $\beta_2 = 0.999$ configuration, you should plan for at least 1,000 warmup steps.

The consequences of getting warmup duration wrong in either direction are asymmetric, which makes this a safety-critical hyperparameter.

If warmup is too short (say, 100 steps when training for 100,000 steps with a large learning rate), the learning rate spikes before Adam has reliable statistics, potentially causing:

• Loss spikes where training destabilizes temporarily and the loss jumps to a much higher value before recovering
• Slow recovery as the optimizer backs away from bad parameter regions
• In severe cases, training divergence that requires a full restart from an earlier checkpoint

If warmup is too long (say, 20% of total training), you lose training efficiency by spending too many steps at low learning rates. The model converges more slowly, and you effectively reduce the average learning rate across training. The cost is not catastrophic but measurable: a training run with 20% warmup may require 10-15% more total compute to reach the same loss as one with 2% warmup.

The sweet spot is the range where warmup is long enough for Adam's statistics to stabilize but short enough that the model spends the majority of its budget training at or near the peak learning rate. In practice, erring slightly on the side of longer warmup is safer than shorter warmup: the efficiency cost of slightly too-long warmup is small, while the cost of slightly too-short warmup can be a diverged training run.