Gradient Clipping › Visualizing Gradient Norms During Training

Plotting gradient norms over training reveals patterns that inform threshold selection. Let's train a model and visualize the gradient dynamics over hundreds of steps to understand what "normal" looks like and what problematic patterns to watch for.

The gradient norm trace is one of the most informative diagnostic plots available for neural network training. Unlike the loss curve, which can decrease smoothly even when the network is on the edge of instability, the gradient norm trace reveals the moment-to-moment health of optimization. A stable training run shows gradient norms fluctuating within a band, neither systematically growing nor shrinking dramatically. Occasional spikes are normal and expected; a sustained upward trend suggests instability.

The gradient norm plot reveals the training dynamics. Early in training, gradients tend to be larger and more variable as the model makes large initial adjustments. As the model converges, gradients stabilize and shrink. The occasional spikes are the dangerous outliers that gradient clipping targets, and the threshold at 2.0 correctly identifies them as the minority of steps where intervention is needed.

In practice, you would generate this plot from real training data rather than a simulation, and you would use it to set your clipping threshold before finalizing your training configuration. A good workflow is to run a short exploratory training session without clipping, plot the gradient norm trace, identify the 90th or 95th percentile, and use that as your threshold. This data-driven approach is more reliable than using a default value like 1.0 without verification.