Gradient Clipping › Choosing the Right Threshold
Language AI HandbookAll chapters Gradient Clipping › Choosing the Right ThresholdOne section · Sections in this chapter7 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Selecting an appropriate clipping threshold requires balancing two concerns: clip too aggressively and you slow down learning; clip too loosely and explosions still occur. The threshold is not a hyperparameter that directly affects what the model learns in the way that learning rate or architecture does. Rather, it is a guard rail that prevents catastrophic failures.
The threshold should be high enough to allow the vast majority of gradients through unmodified. Only the outlier gradients, the ones significantly above the typical range, should trigger clipping. If you set the threshold too low, you are effectively capping the learning rate in a non-uniform way, which can distort optimization and slow convergence. The goal is emergency intervention, not routine regulation.
Empirical Guidelines
Start with a threshold based on observed gradient norms during initial training runs:
- Conservative: Set threshold at the 90th percentile of observed gradient norms. This clips only the most extreme values.
- Moderate: Use the 75th percentile. Clips more frequently but still allows most gradients through unchanged.
- Aggressive: Set threshold at the median. Halves most gradient magnitudes, useful for very unstable training.
def recommend_threshold(gradient_norms, percentile=90):
"""Recommend a clipping threshold based on observed gradient norms."""
threshold = np.percentile(gradient_norms, percentile)
return threshold
# Using our simulated gradient norms
thresholds = {
"Conservative (90th)": recommend_threshold(gradient_norms, 90),
"Moderate (75th)": recommend_threshold(gradient_norms, 75),
"Aggressive (50th)": recommend_threshold(gradient_norms, 50),
}Recommended Clipping Thresholds -------------------------------------------------- Gradient norm statistics: Min: 0.0135 Max: 4.2679 Mean: 0.7562 Std: 0.5029 Threshold recommendations: Conservative (90th): 1.0466 (would clip 10.0% of batches) Moderate (75th): 0.8660 (would clip 25.0% of batches) Aggressive (50th): 0.6914 (would clip 50.0% of batches)
The conservative threshold at the 90th percentile would clip only the most extreme 10% of gradients. This allows normal training dynamics while preventing outlier spikes. The moderate threshold clips about 25% of batches. This provides more aggressive stabilization. The aggressive option clips half of all gradients, useful only for severely unstable training where stability matters more than convergence speed.
Common Threshold Values
Certain threshold values have become standard through empirical practice across many research papers and production systems:
- 1.0: A common default for transformer models. Works well with Adam optimizer and typical learning rates.
- 5.0: More permissive, used when gradients are naturally larger or training is more stable.
- 0.5: Aggressive clipping for very deep or recurrent networks with known stability issues.
The right value depends on your architecture, optimizer, and learning rate. Higher learning rates generally require lower clipping thresholds to prevent large updates. In the original BERT paper, the authors used a max norm of 1.0 throughout training, and this value has propagated into many subsequent transformer implementations as a sensible default. GPT models have similarly adopted values between 1.0 and 5.0 depending on model size and training batch size.
In practice, the threshold also interacts with the batch size. Larger batches produce gradient estimates averaged over more examples, which tends to reduce the variance of gradient magnitudes. If you increase batch size, your gradient norms may decrease on average, potentially allowing a higher threshold. Conversely, if you use a very small batch size (or even single-example updates), expect higher variance in gradient norms and potentially need a lower threshold.