Adam Optimizer › Gradient Noise and Adam's Robustness

One reason Adam dominates practical deep learning is its robustness to gradient noise. When training with small minibatches, the gradient estimate is noisy: each batch samples only a fraction of the training data, and the resulting gradient is a stochastic approximation of the true gradient. This noise has two distinct effects on optimization.

The first effect is noise in gradient direction. On any given step, the gradient might point in a slightly wrong direction due to the random batch composition. Momentum (the first moment $m_t$) mitigates this by averaging over recent steps. If the true gradient points consistently in one direction but is corrupted by zero-mean noise, the average over $1/(1-\beta_1)$ steps will be much closer to the true direction than any individual step.

The second effect is noise in gradient magnitude. Stochastic gradients have variable magnitude even when the direction is correct. A batch that happens to contain many hard examples will produce a large gradient; a batch of easy examples produces a small one. The second moment $v_t$ captures this variability. By dividing by $\sqrt{v_t}$, Adam normalizes away the magnitude variation, making step sizes more consistent even when individual batch gradients vary widely.

This dual smoothing is why Adam with small batches often converges faster in terms of wall-clock time than SGD with large batches. With large batches, SGD has low gradient variance and converges well; with small batches (which process more data per unit time on a single GPU), Adam's noise robustness is decisive. In language model training, where batch sizes are often constrained by memory and gradients are computed from short context windows, this property is particularly valuable.