Momentum › Summary

Momentum converts SGD from a reactive optimizer that takes independent steps into a physics-inspired optimizer that accumulates direction and speed across iterations. The key ideas are:

• Momentum maintains a velocity variable $\mathbf{v}_t$ as an exponential moving average of past gradients, controlled by the coefficient $\beta$. In the PyTorch formulation, this is $\mathbf{v}_t = \beta \mathbf{v}_{t-1} + \nabla \mathcal{L}(\theta_{t-1})$.

• In narrow valley landscapes, momentum dampens oscillations across the steep dimensions while accelerating progress along consistent gradient directions toward the minimum. This behavior is equivalent to a low-pass filter on the gradient signal, suppressing high-frequency oscillation noise while preserving low-frequency trend information.

• The effective learning rate for persistent gradient directions is amplified by a factor of $1/(1-\beta)$, which is 10x for the standard $\beta = 0.9$ setting. This amplification requires reducing the base learning rate by the same factor when switching from plain SGD to SGD with momentum.

• Nesterov momentum evaluates the gradient at a lookahead position (the position after applying the committed velocity). This provides an earlier corrective signal and often achieving faster convergence than classical Polyak momentum. It achieves the good $O(1/t^2)$ convergence rate for convex problems. PyTorch implements it at no computational overhead via algebraic reformulation.

• Momentum interacts with learning rate schedules through its natural warm-up behavior (velocity starts at zero and builds up over $1/(1-\beta)$ steps) and through the need to reset velocity at learning rate restarts (warm restarts require momentum reset for best results).

• The primary limitation of momentum is that it is not adaptive: it applies the same smoothing to all parameters regardless of their individual gradient scales. The next chapter introduces Adam, which addresses this by maintaining per-parameter adaptive scaling in addition to the momentum mechanism covered here.

• Despite Adam's advantages for transformer training, SGD with momentum remains the optimizer of choice for large-scale vision model training, often achieving better generalization than Adam when given sufficient training time. Understanding momentum is therefore essential both as a technique in its own right and as a foundation for understanding the first-moment component of Adam and its variants.