Adam Optimizer › The Adam Update Rule

With bias-corrected moment estimates in hand, the Adam parameter update is:

$$\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$

where:

• $\theta_t$: the parameter vector at step $t$
• $\alpha$: the global learning rate (typically 0.001)
• $\hat{m}_t$: bias-corrected first moment (direction and momentum)
• $\hat{v}_t$: bias-corrected second moment (per-parameter scale)
• $\epsilon$: a small constant for numerical stability, typically $10^{-8}$

The full Adam algorithm, written out step by step, is:

$$\begin{aligned} g_t &= \nabla_\theta \mathcal{L}(\theta_{t-1}) && \text{(compute gradient)} \\ m_t &= \beta_1 m_{t-1} + (1 - \beta_1) g_t && \text{(update first moment)} \\ v_t &= \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 && \text{(update second moment)} \\ \hat{m}_t &= \frac{m_t}{1 - \beta_1^t} && \text{(bias-correct first moment)} \\ \hat{v}_t &= \frac{v_t}{1 - \beta_2^t} && \text{(bias-correct second moment)} \\ \theta_t &= \theta_{t-1} - \frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t && \text{(update parameters)} \end{aligned}$$

where:

• $\mathcal{L}$: the loss function
• All other variables are as defined above

The $\epsilon$ term in the denominator prevents division by zero when $\hat{v}_t \approx 0$, which happens for parameters that have received no gradients. But $\epsilon$ has an additional effect on Adam's behavior that is often overlooked.

Consider a parameter where $\hat{v}_t$ is very small, say $10^{-12}$. Then $\sqrt{\hat{v}_t} + \epsilon \approx \epsilon = 10^{-8}$, and the effective learning rate is $\alpha / \epsilon = 0.001 / 10^{-8} = 100,000$. This is enormous. In practice, if a parameter truly receives near-zero gradients consistently, its $\hat{m}_t$ is also near zero, so the actual update $(\alpha / \epsilon) \cdot \hat{m}_t$ remains small. The two small quantities cancel out.

However, if $\epsilon$ is set too small (e.g., $10^{-16}$), numerical precision issues can cause $\hat{v}_t$ to underflow to zero in floating-point arithmetic, resulting in infs or nans in the updates. If $\epsilon$ is set too large (e.g., 1.0), Adam degenerates toward vanilla SGD because the denominator becomes dominated by $\epsilon$ rather than $\sqrt{\hat{v}_t}$, eliminating the adaptive behavior. The default $\epsilon = 10^{-8}$ is a pragmatic choice that avoids both failure modes in typical float32 computation.

PyTorch's torch.optim.Adam uses $\epsilon = 10^{-8}$ by default but exposes it as a tunable parameter. Some practitioners increase it to $10^{-7}$ or even $10^{-5}$ when training with mixed precision (float16), since smaller values can cause numerical instability at reduced precision.