Gradient Clipping › Gradient Clipping with Different Optimizers
Language AI HandbookAll chapters Gradient Clipping › Gradient Clipping with Different OptimizersOne section · Sections in this chapter9 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The interaction between gradient clipping and optimizers matters and is often misunderstood. Adaptive optimizers like Adam already normalize gradients per-parameter using second-moment estimates, which can reduce the severity of gradient explosions. However, this normalization happens after clipping, and it does not eliminate the need for clipping in the presence of true gradient explosions.
Think of it this way: Adam divides each parameter's gradient by the running estimate of its standard deviation, effectively normalizing the update scale per-parameter. This makes Adam less sensitive to poorly-scaled gradients in general. But if the gradient norm grows by a factor of 1000 due to an explosion, Adam's per-parameter normalization damps this effect somewhat but does not prevent it entirely. The explosion is distributed across all parameters, and while Adam reduces the relative impact, the absolute update magnitudes can still be catastrophically large.
# Comparing gradient clipping needs across optimizers
def analyze_optimizer_gradients(optimizer_class, lr, n_steps=100):
"""Analyze gradient behavior with different optimizers."""
model = nn.Sequential(
nn.Linear(10, 64),
nn.ReLU(),
nn.Linear(64, 64),
nn.ReLU(),
nn.Linear(64, 1),
)
optimizer = optimizer_class(model.parameters(), lr=lr)
gradient_norms = []
update_norms = []
for step in range(n_steps):
x = torch.randn(32, 10)
y = torch.randn(32, 1)
optimizer.zero_grad()
loss = nn.MSELoss()(model(x), y)
loss.backward()
# Record gradient norm
grad_norm = (
sum(
p.grad.norm().item() ** 2
for p in model.parameters()
if p.grad is not None
)
** 0.5
)
gradient_norms.append(grad_norm)
optimizer.step()
return gradient_norms
sgd_norms = analyze_optimizer_gradients(optim.SGD, lr=0.01)
adam_norms = analyze_optimizer_gradients(optim.Adam, lr=0.001)Gradient Norm Comparison Across Optimizers -------------------------------------------------- SGD (lr=0.01): Mean gradient norm: 0.8244 Max gradient norm: 2.0911 Std gradient norm: 0.2990 Coefficient of variation: 0.36 Adam (lr=0.001): Mean gradient norm: 0.8384 Max gradient norm: 2.3709 Std gradient norm: 0.3221 Coefficient of variation: 0.38
Both optimizers show similar gradient norm distributions since we measure gradients before the optimizer transforms them. The coefficient of variation (std/mean) indicates gradient stability. While Adam's adaptive per-parameter scaling helps with the actual updates, it does not change the raw gradient magnitudes that clipping operates on. For very unstable training, combining Adam with gradient clipping provides complementary stability mechanisms: Adam handles routine variation in gradient scale across parameters, while clipping handles the occasional catastrophic spikes that Adam alone cannot tame.
