Learning Rate Warmup › Learning Rate Warmup

Training a large neural network from scratch is a delicate process. The parameters start randomly initialized, the gradients are enormous, and the optimizer has no accumulated history to guide it. If you immediately start training with your target learning rate, those first few updates can be catastrophically large, sending weights into regions of the loss landscape that are hard to recover from. Learning rate warmup is the solution: instead of jumping straight to your intended learning rate, you start very small and gradually increase it over a defined number of steps, giving the model time to settle into a stable training regime before committing to large gradient steps.

Warmup is a practical heuristic with firm theoretical grounding, and it is standard practice in virtually every large-scale language model training run. GPT, BERT, T5, LLaMA, and nearly every transformer trained since 2018 uses some form of learning rate warmup. Understanding why warmup works, how to implement it, and how to tune it is essential knowledge for anyone training neural networks at scale.

Think of warmup as the equivalent of warming up a car engine before driving it hard. The engine works best at operating temperature, not straight from a cold start. Similarly, the optimizer works best when its internal statistics have had a chance to reflect the actual training dynamics of the model, not the chaotic, high-noise conditions of step one. The metaphor extends further: just as a cold engine can handle gentle acceleration but stalls under aggressive throttle, the optimizer at step one can handle small updates safely but collapses under large ones.