Momentum › Dampening Oscillations in Narrow Valleys

To make the oscillation-dampening effect concrete, consider what happens to gradient components in a narrow valley.

Along the valley floor (shallow direction), the gradient consistently points toward the minimum. Each step adds a positive contribution to $\mathbf{v}_t$ in this direction. The velocity in the floor direction accumulates, so the effective step size in this direction grows over time.

Across the valley (steep direction), the gradient alternates sign every step: first positive (pointing left), then negative (pointing right), then positive again. With momentum, these alternating contributions partially cancel in the exponential moving average. If the gradient alternates as $+g, -g, +g, -g, \ldots$, consecutive terms cancel and the velocity in that direction approaches zero.

The net effect is that the optimizer moves quickly toward the minimum along the floor while barely oscillating across the steep walls. More precisely, if a gradient component is consistently $+g$ across many steps, the velocity in that direction builds toward $g/(1-\beta)$ in the PyTorch formulation. If it alternates sign, the contributions cancel and velocity stays near zero.

This asymmetry is the heart of why momentum is effective. The optimizer is not just moving faster: it is moving smarter, allocating effective learning rate capacity to directions where the gradient provides consistent, trustworthy information, and withholding it from directions where the gradient is noisy or oscillatory.

Let us think about this more carefully using the frequency domain analogy. Each parameter direction's gradient, viewed as a function of training step, is a signal. A gradient component that consistently points in one direction is a low-frequency signal (constant, slowly varying). A gradient component that oscillates back and forth every step is a high-frequency signal (alternating). The exponential moving average with coefficient $\beta$ attenuates high-frequency signals and passes low-frequency signals, just like an electronic low-pass filter. The cutoff frequency is approximately $(1-\beta)/(2\pi)$ in normalized units. Lower $\beta$ means higher cutoff frequency, meaning less filtering. Higher $\beta$ means lower cutoff frequency, meaning more aggressive smoothing.

This frequency interpretation reveals something important: momentum is not eliminating noise by averaging multiple samples of the same quantity (as you would in a batch size increase). It is eliminating directional noise by distinguishing between gradient directions that are persistent (signal) and gradient directions that fluctuate (noise). A single sample per step is enough for momentum to work because it exploits temporal correlation across steps rather than spatial averaging across samples.