Gradient Clipping › Summary

Gradient clipping prevents training instability caused by exploding gradients. The technique caps gradient magnitudes before weight updates, keeping optimization on track when backpropagation produces unreasonably large values. It is one of the most widely used stabilization techniques in deep learning, deployed in virtually every large-scale transformer training pipeline.

The key takeaways from this chapter:

• Exploding gradients occur when gradient magnitudes grow exponentially during backpropagation, especially in deep or recurrent architectures. They manifest as NaN losses and training collapse.
• Clip by value constrains each gradient element independently to $[-\tau, \tau]$, where $\tau$ is the clipping threshold. It is simple but can distort gradient direction.
• Clip by global norm rescales the entire gradient vector to have at most norm $\tau$ while preserving relative proportions. This preserves the update direction and is generally preferred.
• PyTorch's clip_grad_norm_ implements global norm clipping and is the standard choice for most applications.
• Threshold selection should be based on observed gradient norm distributions. Common values range from 0.5 to 5.0, with 1.0 being a typical default.
• Monitor clipping frequency to calibrate thresholds. Clipping on 5-20% of batches usually indicates a well-tuned threshold.
• Gradient accumulation requires clipping after all micro-batches are accumulated, not per-micro-batch, to maintain consistent semantics.
• Clipping is a safety mechanism, not an optimizer. Consistent heavy clipping signals an underlying issue with architecture or initialization that should be addressed at its root cause.

Gradient clipping is essential for training RNNs, transformers, and other deep architectures. While it does not improve optimization directly, it prevents the catastrophic failures that would otherwise make training impossible. Combined with careful initialization, appropriate learning rate schedules, and normalization techniques, gradient clipping forms part of the standard toolkit for reliable deep learning at scale.