Adam Optimizer › Adam vs SGD and RMSProp

Adam's design draws from two predecessors, and it helps to understand what each contributes.

Vanilla SGD uses a fixed learning rate:

$$\theta_{t+1} = \theta_t - \alpha g_t$$

SGD is simple and works well when gradients are well-scaled, but it has no memory of past gradients and no ability to adapt its step size per parameter.

Momentum adds a velocity term:

$$\theta_{t+1} = \theta_t - v_t, \quad v_t = \beta v_{t-1} + \alpha g_t$$

Momentum smooths oscillations and builds up speed in consistent gradient directions, which helps escape flat regions and ravines. But it still uses a single global learning rate.

RMSProp adapts the learning rate per parameter using the second moment:

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2, \quad \theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{v_t} + \epsilon} g_t$$

RMSProp gives each parameter its own adaptive step size but applies the raw gradient $g_t$ rather than a momentum-smoothed version. This means the update direction can be noisy when individual gradients are noisy.

Adam combines both improvements:

$$\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$

The numerator $\hat{m}_t$ provides the momentum benefit (smoothed direction), and the denominator $\sqrt{\hat{v}_t} + \epsilon$ provides the adaptive scale benefit (per-parameter step size). The bias correction makes both estimates reliable from the start.

The practical implication of this comparison is that Adam's 2x memory overhead over SGD (two tensors per parameter instead of one) is the price you pay for the two improvements. For most modern deep learning workloads where GPU memory is the binding constraint, this tradeoff is well worth it. The improvement in convergence speed typically more than compensates for the smaller effective batch size you can fit on a GPU. However, at very large model scales (billions of parameters), the memory overhead becomes a substantial engineering challenge, as discussed later in the chapter.