Adam Optimizer › Worked Example: Tracing One Parameter Through 5 Steps

Let's trace a single parameter through five Adam update steps to make the mechanics concrete. Suppose the parameter starts at $\theta_0 = 2.0$, and receives gradients $g = [0.5, 0.4, 0.6, 0.3, 0.5]$ over five steps. We'll use the standard hyperparameters: $\alpha = 0.01$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$.

Initialize: $m_0 = 0$, $v_0 = 0$.

Step 1 ($t=1$, $g_1 = 0.5$):

$$\begin{aligned} m_1 &= 0.9 \cdot 0 + 0.1 \cdot 0.5 = 0.05 \\ v_1 &= 0.999 \cdot 0 + 0.001 \cdot 0.25 = 0.00025 \\ \hat{m}_1 &= 0.05 / (1 - 0.9^1) = 0.05 / 0.1 = 0.5 \\ \hat{v}_1 &= 0.00025 / (1 - 0.999^1) = 0.00025 / 0.001 = 0.25 \\ \theta_1 &= 2.0 - \frac{0.01}{\sqrt{0.25} + 10^{-8}} \cdot 0.5 = 2.0 - \frac{0.01}{0.5} \cdot 0.5 = 2.0 - 0.01 = 1.99 \end{aligned}$$

Notice that after bias correction, $\hat{m}_1 = 0.5 = g_1$ and $\hat{v}_1 = 0.25 = g_1^2$. The bias correction brings the estimates in line with the actual gradient magnitude at step 1, giving an effective step of exactly $\alpha / \sqrt{g_1^2} \cdot g_1 = \alpha \cdot \text{sign}(g_1)$. In other words, early in training the Adam update approximates a step of constant size $\alpha$ in the gradient direction, regardless of gradient magnitude. This is the trust region interpretation of Adam.

Step 2 ($t=2$, $g_2 = 0.4$):

$$\begin{aligned} m_2 &= 0.9 \cdot 0.05 + 0.1 \cdot 0.4 = 0.085 \\ v_2 &= 0.999 \cdot 0.00025 + 0.001 \cdot 0.16 = 0.000409775 \\ \hat{m}_2 &= 0.085 / (1 - 0.81) = 0.085 / 0.19 \approx 0.447 \\ \hat{v}_2 &= 0.000409775 / (1 - 0.998001) \approx 0.000409775 / 0.001999 \approx 0.205 \\ \theta_2 &\approx 1.99 - \frac{0.01}{\sqrt{0.205}} \cdot 0.447 \approx 1.99 - 0.00988 \approx 1.980 \end{aligned}$$

The parameter continues to decrease, with each step sized close to $\alpha = 0.01$. This is characteristic of Adam in early training: the per-parameter normalization makes step sizes roughly equal to the global learning rate $\alpha$, giving predictable training dynamics regardless of gradient scale.

By step 5, the moment estimates have accumulated enough history that the bias corrections become smaller. The first moment $m_t$ is now a smooth average of all five gradients, pointing consistently in the positive direction (since all gradients are positive). The second moment $v_t$ has accumulated a sense of the typical gradient magnitude squared, and the effective step size is settling toward a stable value. This accumulation phase is what makes Adam's convergence so reliable: within a few dozen steps, it has calibrated per-parameter step sizes that remain appropriate throughout the rest of training. If you were to plot the effective learning rate $\alpha / (\sqrt{\hat{v}_t} + \epsilon)$ over time, you would see it start near $\alpha$ during the bias-dominated early steps, then gradually settle as $\hat{v}_t$ stabilizes around the true mean squared gradient. This settling process takes longer for parameters with smaller gradients, because their $v_t$ grows more slowly and the bias correction factor remains significant for more steps.