Training Stability › Limitations and Practical Considerations

Gradient clipping is effective but not a complete solution to training instability. It addresses the symptom (large gradient steps) without necessarily addressing the cause (a difficult loss landscape, a bad data distribution, or suboptimal hyperparameters). A model that requires constant aggressive clipping, say more than 50% of steps hitting the threshold, is signaling a deeper problem that deserves investigation rather than simply continuing to clip. When clipping activates this frequently, the practical effect is that you have reduced the effective learning rate substantially, and it may be more principled to simply lower the nominal learning rate and relax the clip threshold, giving you more transparent control over the training dynamics.

The interaction between gradient clipping and adaptive optimizers deserves careful thought. Adam already scales gradients by an estimate of their recent variance; adding norm clipping on top creates a system where the effective learning rate is constrained by two separate mechanisms. For practitioners, this means that the Adam learning rate and the clipping threshold are not fully independent hyperparameters. Aggressive clipping effectively reduces the learning rate for large-gradient steps, which may require a compensating increase in the base learning rate. This interdependence makes joint tuning more important than tuning each parameter in isolation. A useful mental model is to think of gradient clipping as a "circuit breaker" for outlier batches, not as a substitute for appropriate learning rate scheduling.

Loss spike frequency is also a function of dataset quality in ways that are easy to underestimate. Web-scraped text corpora (used for most large language model pretraining) contain a long tail of extremely unusual documents: very long repetitive sequences, documents consisting entirely of symbols, encoding artifacts, and spam. Even with gradient clipping, these sequences can cause persistent elevation in the loss for many steps after the batch is processed, because the optimizer state carries the signal forward. Data filtering pipelines that remove or downweight outlier documents are a complementary and often underinvested stability technique. A rough heuristic: filtering out the top 0.1% of highest-loss documents from a training corpus can reduce spike frequency without noticeably reducing data diversity. In absolute terms, for a training corpus of 1 trillion tokens, this corresponds to removing about 1 billion tokens from the most problematic documents, which is a very small fraction but disproportionately impacts stability.

There is also a tension between stability and expressiveness that deserves acknowledgment. Many stability techniques, including gradient clipping, weight decay, and conservative learning rate schedules, are forms of regularization that limit how aggressively the model can update in response to any single batch. This is the right tradeoff for most training scenarios. But sometimes the gradient norm is large because the model is making an important learning step from a difficult batch. Clipping that gradient prevents the model from fully capturing that signal. This is one reason why stability and final model quality are sometimes in tension, and why finding the right balance (rather than maximizing stability at all costs) is the practitioner's goal.

Stability techniques scale differently with model size. The interventions in this chapter, gradient clipping, pre-norm architectures, learning rate warmup, and optimizer hyperparameter tuning, work at any model scale. But their relative importance changes with scale. For small models (under 1B parameters), stability is easy to achieve and most runs succeed without careful tuning. For models above 10B parameters, stability becomes the dominant operational concern. Long runs (300B+ tokens) have more exposure to rare outlier batches that can trigger instabilities. Very deep models (48+ layers) are more sensitive to initialization and normalization choices. The techniques in this chapter become more valuable and more necessary as model scale increases, which is why they have received so much attention in the era of large-scale pretraining.

Finally, the monitoring infrastructure outlined in this chapter is not optional for serious training runs. It is easy to set up and dramatically reduces the time to diagnose and fix instabilities when they occur. Gradient norm logs can also be valuable retroactively: by examining the gradient norm history at the time a loss spike began, you can often distinguish between a data problem (single-step spike with no precursor), an optimizer state problem (gradual gradient norm increase before a spike), and an architectural problem (gradient norms that are uniformly extreme from early in training). Each of these root causes has different remedies, and the monitoring record is the evidence that distinguishes them. Treating monitoring as a first-class concern, not an afterthought, is one of the most cost-effective investments you can make in a large training project.