Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gradient Clipping › Gradient Clipping Strategies

Michael BrenndoerferUpdated April 24, 202513 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 73Gradient Clipping › Gradient Clipping Strategies

Reading section 3 of 14 · Gradient Clipping Strategies · 13 minSection 3 of 14 · 13 min

Section 3 of 14 · 13 min read

Gradient Clipping › Gradient Clipping StrategiesOne section · Sections in this chapter3 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Now that we understand why exploding gradients are dangerous, let's examine how to prevent them. The core idea is simple: if gradients become too large, we shrink them before applying the weight update. But "too large" can mean different things, and how we shrink gradients matters for optimization. This leads to two distinct strategies, each with its own geometric interpretation and trade-offs.

The two strategies reflect a fundamental choice in how we define the problem. In machine learning, the gradient is a vector in parameter space, a high-dimensional arrow pointing in the direction of steepest ascent of the loss. When we say a gradient is "too large," we could mean two different things. The distinction between these interpretations is not just mathematical pedantry; it has real consequences for how your model learns.

The Central Question: What Should "Too Large" Mean?Link Copied

When we say a gradient is "too large," we could mean two different things:

Individual elements are extreme: Some specific gradient values are huge, even if others are small
The overall magnitude is extreme: The gradient vector as a whole points too far, even if no single element is unusual

These two interpretations lead to fundamentally different clipping strategies. The first gives us "clip by value," which treats each gradient element independently. The second gives us "clip by global norm," which treats the gradient as a unified vector and scales it uniformly. Understanding this distinction matters, because the choice can affect the magnitude of updates and potentially their direction.

In practice, the choice between these strategies matters most when gradients are uneven across dimensions, which is common in neural networks. Different layers, different weight matrices, different output dimensions: they can all have very different natural gradient magnitudes. A clipping strategy that ignores these differences treats all dimensions equally when they are not, potentially distorting the optimization dynamics in subtle ways.

Clip by Value: Element-wise ConstraintLink Copied

The simplest approach treats each gradient element as an independent quantity to be bounded. If any element exceeds a threshold $\tau$ , we clamp it to that threshold. Elements within bounds remain unchanged.

Think of this geometrically: we are constraining the gradient to lie within a hypercube centered at the origin. In 2D, this is a square; in 3D, a cube; in higher dimensions, a hypercube with sides of length $2\tau$ . The constraint is axis-aligned, meaning it treats each dimension independently without regard for how the dimensions relate to each other.

Clip by Value

Given a gradient vector $\mathbf{g} = [g_1, g_2, \ldots, g_n]$ and a clipping threshold $\tau > 0$ , clip by value applies the following transformation to each element independently:

g_i^{\text{clipped}} = \max(-\tau, \min(\tau, g_i))

where:

$g_i$ : the $i$ -th element of the original gradient vector
$\tau$ : the clipping threshold, defining the maximum allowed absolute value
$g_i^{\text{clipped}}$ : the resulting element, guaranteed to satisfy $|g_i^{\text{clipped}}| \leq \tau$

The $\min(\tau, g_i)$ operation caps positive values at $\tau$ , while $\max(-\tau, \cdot)$ floors negative values at $-\tau$ . Together, they project each component onto the interval $[-\tau, \tau]$ .

The operation is essentially the same as the "hard tanh" or "saturating linear" activation applied to gradients. Values within bounds pass through unchanged. Values outside bounds are snapped to the boundary. There is no interpolation, no scaling: the clipped components simply take the boundary value.

Let's see this in action with a concrete example. We'll create a gradient vector with some moderate values and some extreme outliers, then apply clip by value.

In[8]:

Code

def clip_gradient_by_value(gradients, clip_value):
    """Clip each gradient element to [-clip_value, clip_value]."""
    clipped = []
    for grad in gradients:
        clipped.append(torch.clamp(grad, -clip_value, clip_value))
    return clipped


# Example: gradients with some extreme values
example_grads = torch.tensor([-5.0, -1.0, 0.5, 2.0, 10.0, -15.0])
clip_threshold = 2.0
clipped_grads = torch.clamp(example_grads, -clip_threshold, clip_threshold)

Out[9]:

Console

Clip by Value Example
--------------------------------------------------
Original gradients: [-5.0, -1.0, 0.5, 2.0, 10.0, -15.0]
Clip threshold: ±2.0
Clipped gradients: [-2.0, -1.0, 0.5, 2.0, 2.0, -2.0]

Elements clipped: 3/6
  Index 0: -5.0 → -2.0
  Index 4: 10.0 → 2.0
  Index 5: -15.0 → -2.0

Three of the six elements exceeded the threshold and were clamped. The extreme values at indices 0, 4, and 5 are now bounded to ±2.0, while the moderate values at indices 1, 2, and 3 pass through unchanged.

This asymmetric treatment reveals the core limitation of clip by value: it changes the gradient direction. Before clipping, the gradient might have pointed predominantly toward adjusting parameter 4 (with gradient 10.0). After clipping, that dimension is capped at 2.0, the same as dimension 3. The relative importance of different parameters has been distorted. In optimization terms, we are no longer moving in the direction the loss landscape suggested, but in a direction bent by our clipping constraints.

This direction distortion is not always harmful. In situations where gradients become extreme because of a single unusual data point or a temporary numerical instability, removing that extreme element and letting the other gradients guide the update can be reasonable. But as a general optimization strategy, it introduces a systematic bias: parameters with large natural gradients are systematically under-updated relative to parameters with small natural gradients. Over many training steps, this can slow convergence for the parameters that most need large updates.

Clip by Global Norm: Preserving DirectionLink Copied

The direction distortion problem motivates a different approach. Instead of asking "is each element too large?", we ask "is the gradient vector as a whole too long?" If yes, we scale the entire vector down uniformly, preserving all relative proportions.

Geometrically, this constrains the gradient to lie within a hypersphere of radius $\tau$ centered at the origin. In 2D, this is a circle; in 3D, a sphere. Any gradient landing outside this sphere gets projected back onto its surface by scaling, not by truncating individual components.

Why does this preserve direction? Consider a gradient vector $\mathbf{g}$ with norm $\|\mathbf{g}\|_2 = 10$ when our threshold is $\tau = 5$ . To bring the norm down to 5, we multiply every component by $\frac{5}{10} = 0.5$ . Each component shrinks by the same factor, so the ratios between components remain unchanged. The vector points in exactly the same direction, just with half the length.

Clip by Global Norm

Given a gradient vector $\mathbf{g}$ and a maximum norm threshold $\tau > 0$ , clip by global norm applies the following transformation:

\mathbf{g}^{\text{clipped}} = \begin{cases} \mathbf{g} \cdot \dfrac{\tau}{\|\mathbf{g}\|_2} & \text{if } \|\mathbf{g}\|_2 > \tau \\[6pt] \mathbf{g} & \text{otherwise} \end{cases}

where:

$\mathbf{g}$ : the full gradient vector, treating all parameters as a single concatenated vector
$\|\mathbf{g}\|_2 = \sqrt{\sum_i g_i^2}$ : the L2 (Euclidean) norm, measuring the vector's total magnitude
$\tau$ : the maximum allowed norm (clipping threshold)
$\dfrac{\tau}{\|\mathbf{g}\|_2}$ : the scaling factor, always less than 1 when clipping occurs

After clipping, the resulting gradient satisfies $\|\mathbf{g}^{\text{clipped}}\|_2 \leq \tau$ while maintaining $\dfrac{g_i^{\text{clipped}}}{g_j^{\text{clipped}}} = \dfrac{g_i}{g_j}$ for all pairs of components.

The key insight is that uniform scaling preserves the update direction. If the original gradient suggested parameter A should change twice as much as parameter B, that relationship holds after clipping. We simply take a smaller step in the same direction. This is analogous to reducing your car's speed while keeping the steering wheel in the same position: you slow down, but you continue heading in the same direction. In contrast, clip by value is like randomly turning the steering wheel to avoid exceeding a speed limit on certain axes.

Notice that the formula has a natural threshold behavior: when the gradient norm is already below $\tau$ , the scaling factor $\frac{\tau}{\|\mathbf{g}\|_2}$ exceeds 1, and we do not apply it. The gradient is only modified when it exceeds the threshold, never amplified. This "do nothing when safe, scale when needed" property makes norm clipping a conservative operation that only intervenes when necessary.

Let's implement this and verify that direction is preserved. We'll use two gradient tensors representing different parameters and show that their relative proportions remain unchanged after clipping.

In[10]:

Code

def clip_gradient_by_norm(gradients, max_norm):
    """
    Clip gradients by global L2 norm.
    If the total norm exceeds max_norm, scale all gradients proportionally.
    """
    # Step 1: Compute global norm across all gradients
    total_norm = 0.0
    for grad in gradients:
        total_norm += (grad**2).sum()
    total_norm = total_norm**0.5

    # Step 2: Compute scaling factor
    clip_coef = max_norm / (total_norm + 1e-6)  # Add epsilon for stability

    # Step 3: Apply uniform scaling if needed
    if clip_coef < 1.0:
        return (
            [grad * clip_coef for grad in gradients],
            total_norm.item(),
            clip_coef.item(),
        )
    else:
        return gradients, total_norm.item(), 1.0


# Example: two parameter groups with different magnitudes
grad1 = torch.tensor([3.0, 4.0])  # Norm = sqrt(9+16) = 5
grad2 = torch.tensor([6.0, 8.0])  # Norm = sqrt(36+64) = 10
# Global norm = sqrt(3^2 + 4^2 + 6^2 + 8^2) = sqrt(9+16+36+64) = sqrt(125) ≈ 11.18

max_norm = 5.0
clipped, original_norm, scale = clip_gradient_by_norm([grad1, grad2], max_norm)

The implementation follows three steps:

Compute the global norm by summing squared elements across all gradient tensors, then taking the square root
Calculate the scaling factor as $\frac{\tau}{\|\mathbf{g}\|_2}$ , which will be less than 1 if clipping is needed
Apply uniform scaling to every gradient tensor if the norm exceeds the threshold

Out[11]:

Console

Clip by Global Norm Example
--------------------------------------------------
Gradient 1: [3.0, 4.0] (norm = 5.00)
Gradient 2: [6.0, 8.0] (norm = 10.00)
Global norm: 11.18
Max norm threshold: 5.0

Scale factor: 0.4472
Clipped gradient 1: [1.3416407108306885, 1.7888543605804443]
Clipped gradient 2: [2.683281421661377, 3.5777087211608887]
New global norm: 5.00

Verifying direction is preserved:
  Ratio grad1[0]/grad2[0] before: 0.5000
  Ratio grad1[0]/grad2[0] after: 0.5000

The results confirm the theory. The original global norm of 11.18 exceeded our threshold of 5.0, triggering clipping. Both gradient tensors were scaled by the same factor (approximately 0.45), bringing the global norm to exactly 5.0. Most importantly, the ratio between corresponding elements remains identical: 0.5 before and 0.5 after. The gradient direction is perfectly preserved.

In practice, this is why norm clipping is generally preferred for optimization. When we clip by norm, we are still moving in the direction the loss landscape suggested, just taking a more cautious step. The relative importance of each parameter's update remains unchanged. The optimizer's accumulated knowledge about which parameters should receive larger updates, encoded in the gradient direction, is fully respected.

Comparing the Two Approaches: A Geometric ViewLink Copied

The difference between clip by value and clip by norm becomes crystal clear when we visualize them geometrically. Consider a 2D gradient vector, which we can plot as an arrow in a plane. The two clipping methods impose different constraints on where this arrow can point and how long it can be.

Clip by value constrains the gradient to a square (or hypercube in higher dimensions). The boundary is defined by $|g_1| \leq \tau$ and $|g_2| \leq \tau$ independently. A gradient in the corner of this square, like $(3, 3)$ with threshold $\tau = 2$ , gets clipped to $(2, 2)$ , which changes its direction subtly. But a gradient like $(4, 1)$ with the same threshold becomes $(2, 1)$ , dramatically shifting its angle.

Clip by norm constrains the gradient to a circle (or hypersphere). The boundary is defined by $\sqrt{g_1^2 + g_2^2} \leq \tau$ . Any gradient outside this circle gets scaled toward the origin until it touches the circle's edge, preserving its angle perfectly.

Let's see both methods applied to the same gradient vector:

Out[12]:

Visualization

Vector plot showing original gradient and clip-by-value result pointing in different directions. — Clip by value creates a square constraint region. The original gradient (4, 3) has its first component clipped to 3.0 while the second remains unchanged, rotating the vector away from its original direction and distorting the intended update.

Vector plot showing original gradient and clip-by-norm result pointing in the same direction with different magnitudes. — Clip by norm creates a circular constraint region. Both components of the gradient (4, 3) scale uniformly by the same factor, preserving the original direction while reducing the step size to fit within the norm boundary.

The visualization crystallizes the main insight. On the left, clip by value creates a box (square) constraint. The original gradient $(4, 3)$ lies outside this box, so its first component is clipped to 3. But the second component was already within bounds and remains unchanged. The result is that the clipped vector points at a different angle than the original, pulling the update direction away from where the loss landscape suggested we should go.

On the right, clip by norm creates a circular constraint. The same gradient $(4, 3)$ with norm 5.0 exceeds our threshold of 3.0, so we scale the entire vector by $\frac{3}{5} = 0.6$ . Both components shrink proportionally to $(2.4, 1.8)$ , and the angle is perfectly preserved. We are still heading in the right direction, just taking a smaller step.

This geometric intuition extends to any number of dimensions. In the parameter spaces of neural networks with millions of dimensions, clip by value constrains gradients to a hypercube while clip by norm constrains them to a hypersphere. The direction preservation property of norm clipping becomes even more valuable in high dimensions, where the corners of a hypercube can point in vastly different directions than the original gradient. In very high dimensions, nearly all of the volume of a hypercube is near its corners, meaning clip by value almost always distorts the gradient direction nontrivially.