Adam Optimizer › Bias Correction

There is a subtle but important flaw in the raw moment estimates. Both $m_t$ and $v_t$ are initialized to zero at step $t=0$. This means that early in training, the estimates are biased toward zero. At step $t=1$:

$$m_1 = \beta_1 \cdot 0 + (1 - \beta_1) g_1 = (1 - \beta_1) g_1$$

With $\beta_1 = 0.9$, we have $m_1 = 0.1 \cdot g_1$ even though our best estimate of the mean gradient is $g_1$ itself. The true mean should be $g_1$, but our estimate is only 10% of it.

This zero-initialization bias decays over time. After $t$ steps, the first moment is a weighted sum of past gradients:

$$m_t = (1 - \beta_1) \sum_{i=1}^{t} \beta_1^{t-i} g_i$$

Taking the expectation (assuming $g_i$ are drawn from the same distribution with mean $\mu$):

$$\mathbb{E}[m_t] = \mu \cdot (1 - \beta_1^t)$$

where:

• $\mu$: the true mean gradient
• $\beta_1^t$: the decay factor raised to the $t$-th power
• $(1 - \beta_1^t)$: the bias factor, which starts near 0 and approaches 1 as $t$ grows

The bias factor $(1 - \beta_1^t)$ starts at $1 - 0.9 = 0.1$ at step $t=1$ (meaning the estimate is only 10% of the truth) and approaches 1 as $t \to \infty$ (where it becomes accurate). To correct for this initialization bias, Adam divides each moment estimate by its bias correction factor:

$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$$ $$\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$

where:

• $\hat{m}_t$: the bias-corrected first moment estimate
• $\hat{v}_t$: the bias-corrected second moment estimate
• $1 - \beta_1^t$: correction factor for the first moment at step $t$
• $1 - \beta_2^t$: correction factor for the second moment at step $t$

For $\beta_2 = 0.999$, the second moment bias correction is even more severe early on: $1 - 0.999^1 = 0.001$, so $\hat{v}_1 = v_1 / 0.001 = 1000 \cdot v_1$. Without this correction, the second moment estimate is far too small for the first thousand or so steps, which would cause Adam to take enormous steps with a tiny denominator. The bias correction stabilizes training in these early steps, making Adam's behavior in the first epoch much more predictable.

As training progresses, $\beta_1^t \to 0$ and $\beta_2^t \to 0$, so $\hat{m}_t \to m_t$ and $\hat{v}_t \to v_t$. The correction becomes negligible after enough steps. For $\beta_1 = 0.9$, the first moment correction becomes less than 1% after about $t = 46$ steps. For $\beta_2 = 0.999$, the second moment correction becomes less than 1% after about $t = 4,600$ steps.

Without bias correction, you would observe a characteristic artifact at the start of training: the learning rate would appear enormous for the first few hundred steps, because the second moment estimate $v_t$ would be near zero, making $\alpha / \sqrt{v_t}$ very large. This would cause gradient explosions or wildly oscillating loss values before eventually stabilizing. With bias correction, both moments are immediately calibrated to their true statistical values, and Adam's step sizes remain reasonable from step one. This is especially important for transformer training with learning rate warmup: the warmup schedule deliberately starts with a small learning rate and increases it over the first several thousand steps. During this warmup period, the bias correction is doing exactly its intended job. Both the scheduled learning rate and the moment estimates are warming up together, creating doubly conservative early steps that prevent gradient explosions. The two mechanisms are complementary, not redundant.