Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Momentum › Visualizations

Michael BrenndoerferApril 23, 20255 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 67Momentum › Visualizations

Reading section 10 of 18 · Visualizations · 5 minSection 10 of 18 · 5 min

Section 10 of 18 · 5 min read

Momentum › VisualizationsOne section · Sections in this chapter10 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Out[8]:

Visualization

Line plot showing three training loss curves converging over 100 epochs, with momentum variants well below plain SGD. — Training loss curves for SGD, classical momentum, and Nesterov momentum over 100 epochs on a synthetic regression task. Both momentum variants converge much faster than vanilla SGD, reaching lower loss values within the same number of steps. Nesterov momentum shows a marginally smoother initial descent, consistent with its theoretical advantage in the early training phase.

The loss curve plot reveals the key advantage of momentum: both variants converge much faster than plain SGD. Within the first 20 epochs, the momentum-based optimizers have already reached loss values that plain SGD will not achieve until epoch 60 or later. The curves for classical and Nesterov momentum are close but Nesterov's slightly more responsive correction gives it a small advantage in the early phase of training.

Out[9]:

Visualization

Contour plot of elliptical valley with zigzag optimizer path from upper right to center minimum, without momentum. — Gradient descent trajectory in a narrow valley (curvature ratio 50:1) without momentum. The optimizer oscillates wildly across the steep axis while making slow progress toward the minimum at (0, 0), illustrating the zigzag problem that motivates momentum.

Contour plot of elliptical valley with smooth curved optimizer path to center minimum, with momentum applied. — Gradient descent trajectory in the same narrow valley with momentum (beta=0.9). The accumulated velocity smooths out the lateral oscillations and accelerates progress along the shallow axis, reaching the minimum in far fewer effective steps.

These trajectory plots make the oscillation problem and momentum's solution visually clear. The SGD path (left) zigzags across the valley walls, spending many steps moving laterally rather than toward the minimum. The momentum path (right) quickly aligns with the valley floor and approaches the minimum along a smooth arc. The contour ellipses elongated along the x-axis show the difficulty of the loss surface caused by the high curvature ratio.

Out[10]:

Visualization

Line plot of effective learning rate multiplier vs momentum coefficient beta, showing steep growth as beta approaches 1. — Effective learning rate multiplier as a function of the momentum coefficient beta, computed as 1/(1-beta). At the standard beta=0.9, persistent gradient directions receive a 10x amplification; at beta=0.99, the multiplier reaches 100x. This nonlinear relationship explains why increasing beta requires a proportional reduction in the nominal learning rate to maintain training stability.

The effective learning rate curve shows why momentum choice matters so much. At $\beta = 0.9$ (the default), the effective step size for persistent gradient directions is 10 times the nominal learning rate. This is why practitioners typically reduce the base learning rate when switching from plain SGD to SGD with momentum. The relationship between $\beta$ and the multiplier is nonlinear and accelerates rapidly as $\beta$ approaches 1, which is why the difference between $\beta = 0.95$ and $\beta = 0.99$ is so dramatic in practice.