Training Stability › Training Stability

Training a large language model is not a smooth, monotonic descent toward a good solution. At any moment, loss can spike dramatically, gradients can explode, or training can stall entirely without warning. These instabilities waste compute, corrupt checkpoints, and, in the worst case, require restarting runs from scratch. Understanding why they happen and how to prevent them is one of the most practically important skills in modern deep learning.

This chapter focuses on training stability: the set of phenomena that cause training to behave erratically, the diagnostic signals that reveal problems early, and the techniques that keep training on track. We cover loss spikes and their root causes, gradient norm monitoring as an early warning system, and a toolkit of stabilization techniques from gradient clipping to architectural choices. We also develop a debugging workflow for diagnosing instabilities when they occur.

By the end, you will understand what each technique does and why instability happens in the first place, which makes the solutions feel inevitable rather than arbitrary.