Momentum › Why Plain SGD Struggles

Before diving into the mechanics, it helps to understand the problem momentum is designed to solve. The shortcomings of plain SGD are not immediately obvious if you only consider loss surfaces shaped like a round bowl. On a spherically symmetric loss surface, SGD performs perfectly well: every gradient step points directly toward the minimum, and with the right learning rate, you converge efficiently. The problem surfaces when the loss curvature is highly non-isotropic, meaning it differs dramatically across different directions in parameter space. This distinction between easy and hard loss landscapes is needed to understand why virtually every modern training run uses some form of momentum.

Consider a quadratic bowl whose curvature is much stronger in one direction than the other: a coin-shaped bowl where east-west is very steep but north-south is almost flat. This kind of landscape is extremely common in deep learning. Neural networks with many layers and skip connections often have loss surfaces that look like narrow corridors along many axes simultaneously. Even a simple two-layer network can exhibit curvature ratios of 100:1 or higher between its fastest and slowest-changing weight directions.

Plain SGD takes a step in whatever direction the current gradient points. At a point on the steep sidewall, the gradient is dominated by the steep direction. The step overshoots the valley center, lands on the opposite wall, overshoots back, and so on. The optimizer zigzags laterally while barely progressing along the valley's gentle floor toward the minimum.

This zigzagging behavior is not just inefficient. It actively prevents the use of larger learning rates, because a larger step size makes the sideways oscillation worse. You are forced to use a small learning rate to control lateral noise, which then makes progress along the shallow floor glacially slow. You are essentially caught in a tradeoff: a learning rate large enough to make progress along the gentle axis is too large to avoid oscillation along the steep axis.

To visualize how extreme this can get: if the curvature in the steep direction is 100 times the curvature in the shallow direction, an SGD step that moves one unit along the shallow direction moves 100 units across the steep direction, instantly overshooting the valley center. To keep the lateral step small enough to not overshoot, you must use a learning rate 100 times smaller than you would otherwise use, which makes every step along the shallow direction only 1/100 of a unit. Progress is agonizingly slow.

The fundamental cause of this problem is that a single learning rate must serve all parameter directions simultaneously. It needs to be small enough for the steepest direction and large enough for the shallowest direction, and these two requirements conflict when the curvature ratio is large. SGD has no mechanism to allocate different effective step sizes to different parameter directions, so it is always limited by the worst-case direction.

The ideal behavior is to accumulate momentum in the direction of consistent gradient information (the floor direction) while cancelling out gradient components that flip sign repeatedly (the wall direction). A gradient that pointed left last step and now points right contributes nothing in the accumulated direction, but a gradient that pointed forward both last step and this step compounds, making the optimizer move faster along the consistent axis.

This property emerges naturally from exponential averaging of gradients. If you take a weighted average of many past gradient vectors, the components that consistently agree in sign survive the averaging and amplify each other. The components that alternate sign tend to cancel out. The result is a filtered representation of the gradient that suppresses noise and oscillations while preserving the signal.

That is exactly what momentum achieves. It acts as a low-pass filter on the gradient signal, removing high-frequency oscillations while preserving the low-frequency trend toward the minimum. This filtering perspective is mathematically precise: if you analyze the gradient signal as a time series, momentum is a single-pole infinite impulse response (IIR) low-pass filter with a cutoff frequency determined by the momentum coefficient $\beta$. Higher $\beta$ means more aggressive filtering, cutting out more high-frequency noise but also introducing more lag between a change in the true gradient direction and the optimizer's response.

There is a second pathological landscape that motivates momentum beyond narrow valleys: saddle points. A saddle point is a location where the gradient is zero, but the point is neither a local minimum nor a local maximum. At a saddle point, the loss curves upward in some directions and downward in others.

Saddle points are believed to be far more common than local minima in high-dimensional neural network loss surfaces. The argument comes from random matrix theory: for a critical point to be a local minimum, all eigenvalues of the Hessian must be positive. In a 1,000-dimensional parameter space, the probability that all 1,000 eigenvalues are simultaneously positive approaches zero for generic random functions, while the probability of a saddle point (some positive, some negative eigenvalues) is high.

At a saddle point, plain SGD stalls because the gradient is zero and there is no signal to move. In practice, the gradient is never exactly zero due to floating-point noise and minibatch stochasticity, so SGD does not completely freeze. But it slows dramatically near saddle points, sometimes appearing stuck for hundreds of iterations before escaping. Momentum, by contrast, carries the optimizer through the saddle on the strength of the velocity it has built up approaching the saddle. The ball analogy applies here too: a ball rolling along a flat table does not stop when the slope becomes zero; it coasts forward on its inertia.

This saddle-escaping property is one reason researchers have repeatedly found that momentum significantly reduces training time even on smooth, well-conditioned problems, not just on the narrow valley problems where the oscillation-damping explanation applies most directly.