Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Momentum › Visualizing Velocity Buildup

Michael BrenndoerferApril 23, 20253 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 67Momentum › Visualizing Velocity Buildup

Reading section 16 of 18 · Visualizing Velocity Buildup · 3 minSection 16 of 18 · 3 min

Section 16 of 18 · 3 min read

Momentum › Visualizing Velocity BuildupOne section · Sections in this chapter16 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To complete the intuitive picture, let us visualize how the velocity evolves over the first 50 training steps for a simple one-dimensional optimization problem.

In[11]:

Code

# Simulate momentum velocity buildup on a 1D quadratic: L(x) = x^2
# True gradient at x is 2x; we start at x=5.0


def simulate_momentum_1d(start_x, lr, beta, n_steps, gradient_noise=0.5):
    """Simulate SGD with momentum on L(x) = x^2 with optional gradient noise."""
    x = start_x
    v = 0.0
    xs = [x]
    vs = [v]
    for _ in range(n_steps):
        g = 2 * x + np.random.randn() * gradient_noise
        v = beta * v + g
        x = x - lr * v
        xs.append(x)
        vs.append(v)
    return np.array(xs), np.array(vs)


xs_no_mom, vs_no_mom = simulate_momentum_1d(5.0, lr=0.05, beta=0.0, n_steps=50)
xs_mom09, vs_mom09 = simulate_momentum_1d(5.0, lr=0.05, beta=0.9, n_steps=50)
xs_mom099, vs_mom099 = simulate_momentum_1d(5.0, lr=0.01, beta=0.99, n_steps=50)

Out[12]:

Visualization

Line plot of parameter value over 50 steps for plain SGD and momentum coefficients 0.9 and 0.99. — Parameter trajectory for three momentum settings on a noisy one-dimensional quadratic loss. Without momentum (beta=0), the parameter follows each noisy gradient independently; beta=0.9 converges with mild overshoot, while beta=0.99 uses a reduced learning rate to remain stable.

Line plot of absolute velocity over 50 steps for plain SGD and momentum coefficients 0.9 and 0.99. — Velocity magnitude for the same three momentum settings over 50 steps. Plain SGD tracks the current gradient, while beta=0.9 builds velocity quickly and beta=0.99 builds more gradually but reaches a higher peak, illustrating why stronger momentum requires a smaller nominal learning rate.

The velocity buildup plot is illuminating. For plain SGD ( $\beta = 0$ ), the velocity is simply the current gradient at each step: it fluctuates with the noisy gradient and never accumulates. For $\beta = 0.9$ , the velocity builds up quickly over the first 10 steps and then stabilizes as the optimizer enters the basin around the minimum. For $\beta = 0.99$ , the velocity builds up more slowly but reaches a higher peak, requiring the smaller learning rate to prevent the parameter from overshooting too far.