Adam Optimizer › Adam Convergence Properties

The original Adam paper provides a convergence proof for the online learning setting (non-stationary objectives) under the assumption of convexity. In practice, neural network training is non-convex, and the theoretical guarantees do not directly apply. Nevertheless, several empirical and theoretical observations are useful.

For convex problems, Adam achieves $O(\sqrt{T})$ regret, where $T$ is the number of steps. This is the same asymptotic rate as AdaGrad, though Adam's per-step constants are generally better because the exponential moving average avoids the excessive accumulation that slows AdaGrad in long runs.

For non-convex problems, Adam can fail to converge to a stationary point in theory. A paper by Reddi, Kale, and Kumar (2018) demonstrated specific synthetic examples where Adam oscillates between two solutions rather than converging. This led to the AMSGrad variant (discussed below), though in practice Adam converges reliably on most machine learning tasks.

Adam outperforms SGD most clearly in situations with:

• Sparse features: embedding tables, one-hot inputs, high-dimensional vocabularies. The adaptive per-parameter learning rate is decisive here. Embedding rows that are rarely activated can receive large effective learning rates because their second moment accumulates slowly. This is one reason Adam became dominant for NLP before it became dominant for vision.
• Non-stationary objectives: Adam's exponential weighting means it adapts to changing gradient distributions, which matters in curriculum learning or when the data distribution shifts during training.
• Noisy gradients: the momentum term smooths noise, and the adaptive denominator prevents large oscillations from dominating the update.
• Deep networks: parameters at different depths receive gradients of very different magnitude. Adam compensates for this discrepancy without requiring layer-wise learning rate tuning.
• Fast initial convergence requirements: because Adam normalizes step sizes to be approximately $\alpha$ regardless of gradient scale, it tends to make rapid progress in early training. When compute is limited and you cannot afford to train to convergence, Adam often finds a better solution faster.

Careful research has shown that carefully tuned SGD with momentum can match Adam's generalization performance on image classification benchmarks, and sometimes exceeds it. The intuition is that Adam's adaptive learning rates may allow it to find sharp, narrow minima that generalize less well than the broader minima that SGD tends to find. The sharp minimum hypothesis (from Keskar et al., 2017) suggests that minima discovered through large, consistently-directed updates tend to be broader and generalize better. SGD with a large, slowly decaying learning rate naturally finds such broad minima. Adam's adaptive normalization tends to use more cautious, consistent step sizes that can lock in on narrow minima earlier in training.

However, this tradeoff is highly architecture- and task-dependent. For language modeling and transformer training, Adam or its variants are essentially universal. The datasets are large enough, and the loss landscapes well-behaved enough, that the generalization gap between Adam and SGD rarely manifests.

The practical advice from researchers at Google, OpenAI, and DeepMind converges on: use Adam (or AdamW) by default; switch to carefully-tuned SGD with momentum only if you have the compute budget for extensive hyperparameter search and there is evidence from prior work that SGD generalizes better for your specific task.