Momentum › Momentum

Imagine rolling a ball down a hilly landscape. The ball does not restart from zero speed at every step. It carries the velocity it has built up, accelerating through valleys and slowing on uphill slopes. That physical intuition describes exactly what momentum does for neural network optimization. Where vanilla stochastic gradient descent (SGD) moves by a fixed-size step in the direction of the steepest descent at each iteration, momentum lets the optimizer accumulate a sense of direction and speed across iterations, so it can race through flat regions and sail over small bumps without getting stuck.

We covered SGD in the previous chapter, where we saw that the learning rate controls how far we move on each step. But even a well-tuned learning rate struggles in one common scenario: loss surfaces that are much steeper in one dimension than another. In such narrow valley landscapes, plain SGD oscillates side to side across the narrow dimension while making frustratingly slow progress along the valley floor toward the minimum. Momentum solves this by remembering past gradients and letting the optimizer naturally dampen oscillations while building up speed in persistent directions.

This chapter covers the mathematics of momentum, the intuition behind it, Polyak (classical) momentum versus Nesterov momentum, the effective learning rate phenomenon, how momentum interacts with learning rate schedules, and how to configure momentum in PyTorch. By the end, you will understand why momentum is rarely disabled in modern deep learning training runs, how to choose its hyperparameters in practice, and where its limitations lie relative to adaptive methods like Adam.