Momentum › Geometric Intuition: The Ball and the Bowl

The classical analogy for momentum is a ball rolling down a hilly landscape under gravity and friction. The gradient of the loss function corresponds to the slope the ball experiences at its current position. The velocity of the ball is the momentum variable $\mathbf{v}_t$, and friction is controlled by $(1 - \beta)$.

When the ball rolls into a narrow valley, it naturally builds up speed along the floor because the floor gradient consistently points in one direction. The oscillations perpendicular to the floor cancel themselves out over time because they alternate direction. The ball effectively averages out the lateral oscillations and accelerates down the valley.

The coefficient $\beta$ plays the role of friction. At $\beta = 0$, there is maximum friction; the ball's velocity is reset to the current gradient at every step, which is identical to plain SGD. At $\beta \to 1$, friction disappears; the ball accumulates speed indefinitely and overshoots dramatically. The practical sweet spot is $\beta = 0.9$, which provides enough persistence to dampen oscillations while preventing the ball from running so far that it overshoots the minimum.

This analogy predicts behaviors you will observe in training. When you first start training, the ball is stationary and it takes a few steps to build up speed, just as we observe that momentum optimizers have a slow start before accelerating. When training approaches a minimum, the ball overshoots if it has built up too much speed, which matches the observed tendency of high-$\beta$ momentum to oscillate around minima. When you hit a flat saddle region, a ball already moving does not stop at the saddle, it coasts through; this matches the empirical observation that momentum helps escape saddle points faster than plain SGD.

The analogy also helps you understand what happens in high dimensions. Imagine not a two-dimensional bowl but a 10-million-dimensional loss surface. Each parameter direction has its own independent "slope" and the ball rolls simultaneously in all 10 million directions. In some directions the slope is steep and consistent (carrying the ball quickly toward the minimum along those axes), in some directions it alternates (cancelled out), and in some directions it is nearly flat (the ball barely moves). Momentum's filtering action applies simultaneously in all directions, independently, and the aggregate effect is that the optimizer allocates its effective learning rate budget preferentially to the directions where the gradient signal is most consistent.