Training Stability › Summary

Training stability is an active concern throughout the lifetime of any large model training run. The key ideas from this chapter are:

• Loss spikes arise from high-curvature loss landscape regions, exploding gradients through depth, outlier data batches, and optimizer state corruption. Each cause has a different signature and different remedy.
• Gradient norm monitoring is the most important diagnostic tool. A sudden increase in gradient norm precedes many loss spikes and allows intervention before damage occurs. It is a one-line addition to any training loop.
• Gradient clipping by global norm is the primary stability technique. It preserves gradient direction while bounding step magnitude, and it is standard practice in all large language model training. Common threshold: 1.0.
• Architectural choices matter for stability. Pre-norm transformers are significantly more stable than post-norm for deep models. Residual initialization scaling prevents activation variance from growing with depth.
• Adam hyperparameters affect stability directly. Increasing epsilon from $10^{-8}$ to $10^{-6}$ is often necessary in mixed-precision training. AdamW's decoupled weight decay provides more consistent regularization than L2 regularization in Adam.
• Data quality is an underinvested stability lever. Filtering outlier documents from pretraining corpora reduces spike frequency without reducing diversity.
• Recovery from spikes requires rolling back both model parameters and optimizer state together. Rolling back parameters alone leaves corrupted momentum that causes immediate re-instability.
• Debugging instability requires monitoring infrastructure set up proactively, and a systematic workflow for isolating causes when they occur. Layer-wise gradient analysis is the key tool for architectural debugging.

The next chapter covers mixed-precision training, which introduces its own stability considerations and motivates why the techniques in this chapter are especially critical when working with float16 arithmetic.