Training Stability › The Stability-Quality Tradeoff

Not every large gradient signal is a problem. This is worth exploring in depth because a naive reading of this chapter might suggest that stability should be maximized at all costs, which is incorrect.

Some large gradients are large for good reasons. When a model encounters an unusually difficult example that it is poorly calibrated for, the gradient from that example carries a strong, useful learning signal. Clipping that gradient prevents the model from fully incorporating the lesson.

Consider training a language model on domain-specific scientific text. The model, trained predominantly on general web text, initially assigns very low probability to specialized terminology and technical phrasing. When the training corpus introduces scientific papers, the gradients from those documents are large, showing the model's large prediction error. These large gradients are not errors; they are the mechanism by which the model learns the new domain. Aggressively clipping them slows this adaptation.

This is one reason why the optimal clipping threshold is not zero, and why models trained with no clipping (if they can train stably) sometimes outperform models trained with aggressive clipping on in-distribution test sets, while being worse on the out-of-distribution metrics. The stability-accuracy frontier depends on the data distribution and the model's starting point.

Standard gradient clipping applies a hard constraint: gradients above the threshold are rescaled to exactly the threshold. A softer alternative applies a smooth nonlinear transformation that preserves small gradients exactly, applies light smoothing to moderate gradients, and more aggressively bounds large gradients. One such formulation uses:

$$\hat{g}_i = g_i \cdot \frac{c}{\max(c, \|\mathbf{g}\|)}$$

which is just the standard clipping formula, but the key insight is that the "smoothness" can be achieved by using a smaller clip threshold in combination with a higher learning rate. The net effect is equivalent to using a larger clip threshold with a lower learning rate, but the framing as soft clipping makes the tradeoff explicit.

In practice, practitioners rarely use formulations more exotic than standard global norm clipping, because the tuning cost of additional hyperparameters rarely pays off. Standard clipping with a well-calibrated threshold and monitored clip frequency is sufficient for almost all applications.

Not every spike requires intervention. Small spikes (2-3x the rolling average) in the middle of a long stable training run often resolve themselves within 10-20 steps as the optimizer's momentum returns to normal. Intervening with a checkpoint rollback for every small fluctuation is unnecessarily disruptive and wastes compute.

A reasonable policy for a practical training system is:

• Small spikes (loss 2-5x rolling average for 1-3 steps): monitor, no intervention
• Medium spikes (loss 5-20x rolling average or persisting more than 10 steps): reduce learning rate by 10-20% for 500 steps, then resume
• Large spikes (loss 20x+ rolling average or divergence to infinity): checkpoint rollback with optimizer state reset

This tiered approach avoids the overhead of frequent rollbacks while ensuring that catastrophic instabilities are addressed before they waste significant compute. The thresholds depend on model size and training stage; early in training, higher variance is normal and intervention thresholds should be set looser.