Training Stability › Understanding Loss Spike Anatomy: A Numerical Walkthrough

To build deeper intuition, it is worth walking through the exact sequence of events that happens at the parameter level during a loss spike. This makes the abstract mechanisms concrete and clarifies why the interventions in this chapter are effective.

A loss spike is not instantaneous; it unfolds over several steps. Understanding each phase clarifies what can be done at each point.

Phase 1: The trigger step. A batch arrives with unusually high loss. This could be an outlier document, a sharp curvature cliff in the landscape, or a sudden change in the effective learning rate. The gradient computed from this batch has a norm that is much larger than the running average, say 50x. The optimizer applies this gradient to all parameters.

Phase 2: Parameter displacement. After the large update, model parameters are in a different region of the loss landscape than they were a step earlier. If the update was large enough, they have moved past a local minimum and are now in a high-loss region. This is when the loss curve shows the spike: the next batch sees a higher loss than before the trigger.

Phase 3: Optimizer state corruption. Even if the parameters are eventually restored (by gradient descent pulling them back toward a good region), the Adam optimizer's second moment estimate $v_t$ has absorbed a large gradient squared value. Because $\beta_2 = 0.999$, this inflated estimate decays slowly:

$$v_{t+k} = \beta_2^k v_t + (1 - \beta_2^k) \bar{v}$$

where $\bar{v}$ is the typical second moment value. With $\beta_2 = 0.999$ and a typical training run, the spike's influence on $v_t$ is not halved until about 693 steps later ($k$ such that $0.999^k = 0.5$). During those 693 steps, Adam's adaptive step size for the affected parameters is artificially suppressed, because the denominator $\sqrt{\hat{v}_t} + \epsilon$ is larger than it should be. The model is effectively frozen in those parameters while they slowly "forget" the spike.

Let us trace through the numbers for a single parameter to make this vivid. Suppose a parameter $\theta$ has:

• Current value: $\theta_t = 0.5$
• Typical gradient: $g \sim 0.1$ (normal)
• Adam state before spike: $m_t = 0.1$, $v_t = 0.01$ (so $\sqrt{v_t} = 0.1$)
• Learning rate: $\eta = 0.001$, $\epsilon = 10^{-8}$

The normal update magnitude is:

$$\Delta\theta = \frac{\eta}{\sqrt{v_t} + \epsilon} \cdot m_t \approx \frac{0.001}{0.1} \cdot 0.1 = 0.001$$

That is a small, reasonable update. Now a spike batch arrives with gradient $g_{\text{spike}} = 5.0$ (50x normal). After one Adam step:

$$\begin{aligned} m_{t+1} &= 0.9 \times 0.1 + 0.1 \times 5.0 = 0.59 \\ v_{t+1} &= 0.999 \times 0.01 + 0.001 \times 25.0 = 0.034899 \end{aligned}$$

The update magnitude at step $t+1$ is:

$$\Delta\theta_{t+1} = \frac{0.001}{\sqrt{0.034899}} \cdot 0.59 \approx \frac{0.001}{0.1868} \cdot 0.59 \approx 0.00316$$

That is 3.16x larger than the normal update: the momentum-inflated first moment drives a much larger step. Three hundred steps later, the second moment has decayed to approximately:

$$v_{t+300} \approx 0.999^{300} \times 0.034899 + (1 - 0.999^{300}) \times 0.01 \approx 0.027 + 0.0026 \approx 0.0296$$

So even 300 steps after the spike, $v_t$ is still 3x its pre-spike value, and the adaptive step size for this parameter is still suppressed by a factor of $\sqrt{3} \approx 1.73$. The parameter is learning at 58% of its normal rate for 300+ steps after a single bad gradient. At scale, with thousands of affected parameters, this explains the "loss plateau" that practitioners often observe after a spike even when the loss appears to have recovered.

Gradient clipping prevents all of this from happening. By capping $g_{\text{spike}}$ at 1.0 (assuming a clip threshold of 1.0), the spike's contribution to $m_t$ and $v_t$ is:

$$\begin{aligned} m_{t+1} &= 0.9 \times 0.1 + 0.1 \times 1.0 = 0.19 \\ v_{t+1} &= 0.999 \times 0.01 + 0.001 \times 1.0 = 0.011 \end{aligned}$$

This is only slightly above normal. The optimizer state remains well-behaved, and training continues smoothly after the clipped step.