Momentum › Momentum vs. Adam: When to Use Which

A natural question arises after learning about momentum: if Adam (which combines momentum with adaptive per-parameter learning rates) is available and generally works well, why would you ever use SGD with momentum?

The answer comes down to generalization and compute budget. A growing body of empirical evidence suggests that while Adam converges faster (reaches a low training loss in fewer epochs), SGD with momentum often reaches a lower final test error when given enough training time. This phenomenon is particularly well-documented in computer vision training on ImageNet: the best-performing ResNet and ViT models in the literature are typically trained with SGD plus momentum, not Adam. The intuition is that SGD's less aggressive adaptivity causes it to find wider, flatter minima (often called "sharp" vs. "flat" minima in the generalization literature), and flat minima generalize better to test data.

For natural language processing and transformer-based models, the situation is different. Transformers have highly heterogeneous gradient scales across their layers (embedding layers, attention projections, and feed-forward layers all receive gradients of very different magnitudes), and Adam's per-parameter scaling handles this heterogeneity much better. In practice, virtually all transformer training from BERT to GPT to modern LLMs uses Adam or its variants (AdamW, Adan, etc.) rather than SGD with momentum. The adaptive scaling is necessary in practice rather than optional.

The rule of thumb in modern deep learning is:

• For convolutional neural networks and residual networks on vision tasks with large compute budgets: SGD with momentum is the gold standard.
• For transformer-based language models and multi-modal models: Adam/AdamW with momentum (both classical and second-moment) is the standard.
• For quick prototyping and problem types where you are uncertain: start with Adam, then switch to SGD with momentum if generalization is a concern and you have the compute to train longer.

Understanding momentum thoroughly is essential even when you use Adam, because Adam incorporates a first-moment estimate that is exactly the momentum mechanism described in this chapter, combined with a second-moment estimate for adaptive scaling. The momentum component of Adam is initialized and configured with the same beta1 hyperparameter that corresponds to $\beta$ in classical momentum. Knowing how momentum works makes Adam's hyperparameters interpretable rather than opaque.