Training Stability › Gradient Clipping
Language AI HandbookAll chapters Training Stability › Gradient ClippingOne section · Sections in this chapter4 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Gradient clipping is the most widely deployed stability technique in large model training. The idea is straightforward: if the gradient norm exceeds a threshold, rescale all gradients so the norm equals the threshold.
How Gradient Clipping Works
Given a maximum allowed norm , gradient clipping transforms the gradient vector as follows:
where:
- : the raw gradient vector computed by backpropagation
- : the maximum allowed gradient norm (a hyperparameter, typically 1.0)
- : the clipped gradient used for the parameter update
- : the unit vector in the gradient direction, preserving directional information
The critical property of this transformation is that it preserves the direction of the gradient while constraining its magnitude. When the gradient is already within the threshold, nothing changes. When it exceeds the threshold, the optimizer still moves in the correct direction but takes a smaller step. No information about which direction to update is lost; only the step size is bounded.
This is the essential difference between gradient clipping and simply capping individual gradient components at some value. Clipping by individual component changes direction: a large component in one dimension gets reduced while others do not, which can send the optimizer in a substantially different direction. Clipping by global norm maintains the optimizer's directional intent.
Consider a gradient vector with norm and a clip threshold of . Global norm clipping rescales to , which points in exactly the same direction as the original gradient. Component clipping would produce , which is at 45 degrees and points in a completely different direction. This is why global norm clipping is always preferred over component-wise clipping.
Historical Context
Gradient clipping for recurrent networks was formalized by Pascanu, Mikolov, and Bengio in a 2013 paper titled "On the difficulty of training recurrent neural networks." They showed analytically that the loss field of RNNs has cliff-like curvature in directions corresponding to exploding gradients, and that clipping is the appropriate response. The main insight from that paper was that the cliff structure is predictable: it occurs when gradients follow a direction that corresponds to the largest eigenvector of the Hessian, and clipping prevents the optimizer from taking a catastrophically large step in that direction.
The technique was subsequently adopted for transformer training and has become standard in virtually every large language model training codebase, from GPT-2 through GPT-4, LLaMA, and beyond.
Choosing the Clipping Threshold
The clipping threshold is a hyperparameter that requires empirical tuning. Common values are 1.0 for most tasks and 0.5 for particularly unstable setups. The choice involves a tradeoff: a threshold that is too small clips gradients even when they are legitimate and informative, slowing convergence. A threshold that is too large provides no protection against spikes.
A practical approach is to monitor the gradient norm for the first few thousand steps with no clipping (or with a very loose threshold of 100.0) and observe the typical range. Setting the threshold at roughly the 95th percentile of the observed distribution allows normal gradients to pass through unmodified while clipping the true outliers.
You can also tune the clipping threshold during training by observing what fraction of steps are clipped. If fewer than 5% of steps are clipped, the threshold is not restricting legitimate gradients. If more than 20-30% of steps are clipped, the threshold may be too aggressive and could be impeding learning. A well-calibrated threshold clips only the outlier tail, leaving normal gradient dynamics untouched.
# Demonstrate gradient clipping effect
torch.manual_seed(42)
model_clipped = SimpleTransformerBlock(d_model=128, n_heads=4)
model_unclipped = SimpleTransformerBlock(d_model=128, n_heads=4)
# Copy weights so both start identical
model_unclipped.load_state_dict(model_clipped.state_dict())
opt_clipped = optim.AdamW(model_clipped.parameters(), lr=1e-3)
opt_unclipped = optim.AdamW(model_unclipped.parameters(), lr=1e-3)
clip_threshold = 1.0
clipped_norms = []
unclipped_norms = []
for step in range(80):
x = torch.randn(16, 32, 128)
target = torch.randn(16, 32, 128)
# Unclipped model
opt_unclipped.zero_grad()
loss_u = criterion(model_unclipped(x), target)
loss_u.backward()
raw_norm = torch.nn.utils.clip_grad_norm_(
model_unclipped.parameters(), max_norm=float("inf")
)
unclipped_norms.append(raw_norm.item())
opt_unclipped.step()
# Clipped model (same batch)
opt_clipped.zero_grad()
loss_c = criterion(model_clipped(x), target)
loss_c.backward()
clipped_norm = torch.nn.utils.clip_grad_norm_(
model_clipped.parameters(), max_norm=clip_threshold
)
clipped_norms.append(clipped_norm.item())
opt_clipped.step()Steps where clipping activated: 0 of 80 (0.0%) Mean raw gradient norm: 0.1944 Max raw gradient norm: 0.2130 Mean clipped gradient norm: 0.1944
The fraction of steps where clipping activates tells you whether your threshold is well-calibrated. If clipping activates on more than 20-30% of steps, the threshold is too aggressive and may be impeding learning. In this run it never activates because every pre-clipping norm is well below 1.0, so the example illustrates the pass-through case rather than an actual clipping event.
