Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Gradient Clipping › Worked Example: Gradient Clipping in a Two-Parameter Network

Michael BrenndoerferUpdated April 24, 20253 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 73Gradient Clipping › Worked Example: Gradient Clipping in a Two-Parameter Network

Reading section 4 of 14 · Worked Example: Gradient Clipping in a Two-Parameter Network · 3 minSection 4 of 14 · 3 min

Section 4 of 14 · 3 min read

Gradient Clipping › Worked Example: Gradient Clipping in a Two-Parameter NetworkOne section · Sections in this chapter4 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To solidify the mechanics, let's work through a concrete example with a tiny network where we can track every number. Consider a network with exactly two scalar parameters, $w_1$ and $w_2$ , and a simple quadratic loss. This stripped-down scenario lets us see exactly how each clipping method transforms the gradients and what effect that has on the parameter update.

Suppose after a forward and backward pass, we compute:

\frac{\partial L}{\partial w_1} = 8.0 \quad \text{and} \quad \frac{\partial L}{\partial w_2} = 6.0

The gradient vector is $\mathbf{g} = [8.0, 6.0]$ , with global norm:

\|\mathbf{g}\|_2 = \sqrt{8^2 + 6^2} = \sqrt{64 + 36} = \sqrt{100} = 10.0

We will apply both clipping methods with threshold $\tau = 5.0$ and compare the results.

Clip by value with $\tau = 5.0$ : Each component is clamped independently to $[-5, 5]$ .

g_1^{\text{clipped}} = \min(5, 8) = 5.0 \quad \text{and} \quad g_2^{\text{clipped}} = \min(5, 6) = 5.0

The clipped gradient is $[5.0, 5.0]$ with norm $\sqrt{25 + 25} = 5\sqrt{2} \approx 7.07$ . Notice what happened: the original gradient had $g_1 > g_2$ , suggesting that $w_1$ should receive a larger update. After clipping, both parameters receive the same update magnitude of 5.0. The information that $w_1$ needed a bigger adjustment than $w_2$ has been discarded.

Clip by norm with $\tau = 5.0$ : The entire gradient is scaled by $\frac{5.0}{10.0} = 0.5$ .

g_1^{\text{clipped}} = 8.0 \times 0.5 = 4.0 \quad \text{and} \quad g_2^{\text{clipped}} = 6.0 \times 0.5 = 3.0

The clipped gradient is $[4.0, 3.0]$ with norm $\sqrt{16 + 9} = \sqrt{25} = 5.0$ . The ratio $g_1 / g_2 = 4.0 / 3.0 \approx 1.33$ is the same as the original ratio $8.0 / 6.0 \approx 1.33$ . The direction is perfectly preserved, and the magnitude is exactly at the threshold.

In[13]:

Code

import torch

# Two-parameter gradient example
g = torch.tensor([8.0, 6.0])
tau = 5.0

# Clip by value
g_value = torch.clamp(g, -tau, tau)

# Clip by norm
g_norm_val = g.norm(p=2).item()
if g_norm_val > tau:
    g_norm_clipped = g * (tau / g_norm_val)
else:
    g_norm_clipped = g.clone()

Out[14]:

Console

Worked Example: Two-Parameter Gradient Clipping
=======================================================

Original gradient:   [8.0, 6.0]
Original norm:       10.0000
Threshold tau:       5.0

Clip by Value:
  Clipped gradient:  [5.0, 5.0]
  Resulting norm:    7.0711
  Ratio g1/g2 before: 1.3333
  Ratio g1/g2 after:  1.0000
  Direction preserved? No

Clip by Norm:
  Scale factor:      0.5000
  Clipped gradient:  [4.0, 3.0]
  Resulting norm:    5.0000
  Ratio g1/g2 before: 1.3333
  Ratio g1/g2 after:  1.3333
  Direction preserved? Yes

The output confirms the theoretical analysis. Clip by value modifies the gradient ratio from 1.33 to 1.00, erasing the relative information. Clip by norm maintains the exact ratio, preserving the directional intent of the original gradient while reducing its magnitude to the threshold.

This example, while small, captures the essential behavior in networks with millions of parameters. The gradient vector in a real network encodes the relative importance of updating every weight, and norm clipping respects that encoding while imposing a safe step size limit.