Momentum › Limitations and Practical Considerations

Momentum works across many training settings, but it has limitations that practitioners encounter. Understanding these limitations helps you avoid common training failures and decide when to use a different optimizer.

The accumulated velocity that makes momentum effective in flat regions can cause it to overshoot when the loss surface transitions from a flat plateau to a sharp minimum. The optimizer is "moving too fast" as it approaches the minimum and carries past it. This is more pronounced at higher $\beta$ values and can manifest as oscillations around the minimum or even divergence if the minimum is particularly sharp.

Practical mitigation strategies include using a learning rate schedule that reduces the learning rate as training progresses (so the effective step size shrinks near the end), using Nesterov momentum (which applies a corrective force earlier), or reducing $\beta$ late in training. Modern practice often combines all three: start with relatively high $\beta$, schedule the learning rate down, and trust that the reduced learning rate controls the overshoot.

The overshoot problem becomes more severe when the learning rate schedule decreases very rapidly near the end of training. If the learning rate drops by 10x in the final few epochs (a common warm restart strategy), the optimal final position may be much closer to where you started those epochs. Momentum from the early part of the cycle may carry you past the new, closer target. In this regime, resetting the velocity (as described earlier in the warm restart section) combined with the learning rate reset is the safest approach.

Momentum adds one more hyperparameter to tune. The interaction between $\beta$ and the learning rate means that tuning them independently is suboptimal; they should be considered together. The rule of thumb that "effective learning rate = lr / (1 - beta)" helps, but the actual optimization dynamics are more complex, especially on non-quadratic loss surfaces where the curvature changes as the optimizer moves.

A practical approach is to fix $\beta = 0.9$ and only tune the learning rate. This reduces the hyperparameter search to a single dimension while still getting most of the benefit of momentum. If the training loss is noisy, try increasing $\beta$ to 0.95 while halving the learning rate. If you have a learning rate schedule with a large range (initial LR much larger than final LR), be aware that the effective step size amplification is the same at all learning rates, which can make the early part of training disproportionately aggressive.

A fundamental limitation of momentum is that it uses the same velocity smoothing for every parameter. A parameter with a very large gradient and a parameter with a tiny gradient both get the same $\beta$. This means momentum does not adapt to the scale or curvature of individual parameter directions.

In practice, parameters in different layers of a neural network can have wildly different gradient scales. The input embedding layer may receive gradients 100 times smaller than the output layer. Using a single $\beta$ for all parameters means the momentum is calibrated for some parameters but not others. The Adam optimizer, which we cover in the next chapter, addresses this by maintaining per-parameter running statistics that normalize the update scale. This is one of the primary reasons Adam has largely replaced SGD-with-momentum as the default optimizer for many deep learning applications, particularly transformer-based models.

The non-adaptive limitation also means that momentum does not automatically adjust to changes in loss landscape geometry during training. As the optimizer moves from the broad basin of a loss surface into a narrow valley, the appropriate learning rate for each direction changes. Momentum does not detect this shift and must rely on the user's learning rate schedule to compensate. Adaptive methods effectively implement a per-direction, time-varying learning rate schedule automatically, which is a significant advantage in practice.

If the data distribution shifts during training (as in online learning or curriculum learning), the accumulated velocity may push the optimizer in directions that were useful for the old distribution but are harmful for the new one. The momentum "remembers" gradients from data the optimizer is no longer training on, which can cause it to move in the wrong direction for several steps after the shift. Resetting the velocity when the data distribution changes can help in these scenarios.

In curriculum learning, where the training data is presented in a carefully designed order from easy to hard examples, this issue is particularly relevant. The gradients from easy examples early in training may build up velocity in directions that are precisely opposite to what hard examples require later. Managing the velocity reset between curriculum stages is a practical consideration that is easy to overlook.

The effective gradient noise decreases as batch size increases (by the law of large numbers, larger batches give more accurate gradient estimates). When you scale up batch size by a factor of $k$, the gradient noise variance decreases by $k$. This changes the optimal momentum coefficient: with lower noise, you need less smoothing, and you can often use a slightly lower $\beta$ (or equivalently, a slightly higher effective learning rate) when training with large batches.

The "linear scaling rule" for batch sizes (multiply the learning rate by $k$ when multiplying the batch size by $k$) was developed for plain SGD but applies to momentum as well, with the caveat that the warm-up period is especially important at large batch sizes because the initial gradient estimates are very accurate and the natural warm-up from zero velocity is less beneficial.

Despite these limitations, SGD with momentum remains the optimizer of choice for large-scale computer vision training and many language model fine-tuning workflows. Its simplicity, predictability, and well-understood behavior make it a reliable baseline that Adam-based methods must beat to justify their additional complexity. The ImageNet training recipes that achieve state-of-the-art results with ResNets and ViTs predominantly use SGD with momentum, often achieving lower final error than Adam variants when the training budget is sufficient. Momentum is not being retired any time soon.