Adam Optimizer › The Two Moment Estimates

Adam names its running statistics after statistical moments. Recall that for a random variable $g$:

• The first moment is the mean: $\mathbb{E}[g]$
• The second moment is the mean of the square: $\mathbb{E}[g^2]$ (uncentered variance)

Adam does not compute these exactly. Instead it maintains exponential moving averages that approximate them. Let $g_t$ be the gradient of the loss with respect to a single parameter at step $t$. Adam maintains:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$

where:

• $m_t$: the first moment estimate (exponential moving average of gradients)
• $v_t$: the second moment estimate (exponential moving average of squared gradients)
• $\beta_1$: the decay rate for the first moment, typically 0.9
• $\beta_2$: the decay rate for the second moment, typically 0.999
• $g_t$: the gradient at time step $t$

The first moment $m_t$ is recognizable as the velocity term from momentum. When gradients consistently point in one direction, $m_t$ builds up in that direction. When gradients oscillate, contributions cancel out. This is exactly the momentum behavior we saw in Chapter 7.

The second moment $v_t$ tracks the recent average of squared gradients. If a parameter has been receiving large gradients (either positive or negative), $v_t$ will be large. If it has received small or near-zero gradients, $v_t$ will be small. Squaring keeps $v_t$ positive and captures magnitude regardless of sign.

The Adam update will divide the first moment by the square root of the second moment. To see why this achieves adaptive learning rates, consider two parameters:

Parameter A (dense gradients): receives gradients like $[0.8, 0.7, 0.9, 0.8, ...]$ repeatedly. The second moment $v_t \approx 0.64$. The effective learning rate is scaled by $1/\sqrt{0.64} \approx 1.25$. Not much amplification because this parameter is already getting strong updates.

Parameter B (sparse gradients): receives gradients like $[0.01, 0.0, 0.0, 0.0, 0.5, 0.0, ...]$ rarely. The second moment $v_t \approx 0.002$ (most steps contribute zero). The effective learning rate is scaled by $1/\sqrt{0.002} \approx 22$. This rare-but-important parameter gets a much larger effective step.

The division by $\sqrt{v_t}$ naturally equalizes the learning pace: parameters with large historical gradients get a small effective learning rate, and parameters with small historical gradients get a large effective learning rate. The optimizer finds a reasonable pace for each parameter without any manual tuning.

There is a useful geometric interpretation of what Adam is doing. The second moment estimate $\sqrt{v_t}$ approximates the root-mean-square gradient for each parameter. Dividing by this quantity normalizes the update so that each parameter receives a step whose size is roughly proportional to the inverse of its typical gradient magnitude. In a sense, Adam is performing a diagonal approximation to the natural gradient: it rescales the gradient to account for the curvature of the loss surface along each parameter axis independently. Full natural gradient methods (like K-FAC) use the full Fisher information matrix to perform this rescaling, which is computationally prohibitive at large scale. Adam's diagonal approximation is cheap to compute and surprisingly effective in practice, capturing the most important per-axis curvature information while staying computationally tractable.