Gradient Clipping › Gradient Clipping

Deep neural networks are powerful function approximators, but training them can be surprisingly fragile. One moment your loss is decreasing steadily, the next it explodes to infinity. The culprit is exploding gradients. When gradients grow too large during backpropagation, weight updates become catastrophically destabilizing, and training collapses. The parameters jump to extreme values, and from those extreme values, the network often never recovers.

Gradient clipping provides a simple but effective solution. By capping gradients at a maximum threshold before applying updates, we prevent the runaway feedback loops that cause explosions. This technique is essential for training recurrent neural networks, transformers, and many other deep architectures. Without it, models with long computational paths would be nearly impossible to train reliably.

Think of gradient clipping as a speed limiter for parameter updates. The loss landscape is the terrain your optimizer traverses, and gradients are the slopes you follow downhill. On most of the terrain, the slopes are gentle and you make steady progress. But occasionally you hit a cliff, a region where the loss function drops precipitously, creating an enormous gradient. Without intervention, you'd launch off that cliff and land somewhere completely random. Clipping prevents this by ensuring you never take more than a controlled step size, even on the steepest terrain.

The technique was popularized in a highly influential 2013 paper by Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio, who provided both the theoretical analysis of why recurrent networks suffer from gradient explosions and the practical prescription for addressing them. Their insight was that the gradient explosion problem, while alarming in its consequences, is fundamentally geometric: gradients point in the right direction but are unreasonably long. The fix, therefore, is not to change the direction but to limit the length.

This chapter covers how to detect exploding gradients, the two main clipping strategies (by value and by global norm), and when to apply each approach. We will implement gradient clipping from scratch, explore the geometry of both methods through visualization, and develop practical guidance for choosing appropriate thresholds through gradient norm monitoring. By the end, you will understand how clipping works mechanically, why it is designed the way it is, and how to use it effectively in your own training pipelines.