Adam Optimizer › Adam Hyperparameters

Adam has four hyperparameters: $\alpha$, $\beta_1$, $\beta_2$, and $\epsilon$. Understanding what each controls helps you diagnose training problems.

The learning rate remains the most important hyperparameter. Adam's default of $\alpha = 10^{-3}$ works for a surprisingly wide range of tasks, but you should expect to tune it. Rules of thumb:

• Start with $\alpha = 10^{-3}$ for new problems
• If training loss explodes or diverges immediately: reduce by 10x
• If training is too slow after 10% of planned epochs: increase by 3-10x
• Use a learning rate finder (warm-up scan) to identify the range of valid learning rates

One important difference from SGD: Adam's effective per-parameter learning rate is $\alpha / (\sqrt{\hat{v}_t} + \epsilon)$. When you change the global $\alpha$, you are scaling all per-parameter learning rates proportionally. The adaptive normalization means Adam is generally less sensitive to the precise choice of $\alpha$ than SGD is, but the relative magnitudes still matter.

$\beta_1$ controls how much past gradients influence the current update direction. With $\beta_1 = 0.9$ (default), the effective momentum window is approximately $1 / (1 - 0.9) = 10$ steps. This means Adam averages gradient direction over the last ~10 steps.

Lowering $\beta_1$ makes Adam respond more quickly to recent gradient changes, which can help in non-stationary settings but increases noise. The default 0.9 is almost never worth changing. Some practitioners use $\beta_1 = 0$ (effectively no momentum) for specific research experiments to isolate the adaptive learning rate effect, but this is rarely beneficial in practice.

$\beta_2$ controls how long Adam remembers past gradient magnitudes. With $\beta_2 = 0.999$ (default), the effective window is $1 / (1 - 0.999) = 1000$ steps. The second moment tracks a much longer history than the first moment.

This asymmetry is intentional: you want the denominator to be a stable estimate of the gradient's typical magnitude, not one that jumps around with each new batch. However, $\beta_2 = 0.999$ can be slow to adapt if the gradient scale changes dramatically during training (for example, when learning rate warm-up causes a large jump in gradient norms). In such cases, $\beta_2 = 0.99$ or even $0.98$ can respond faster.

In transformer training, $\beta_2 = 0.98$ has been found to work better than the default 0.999 in some settings (the original "Attention Is All You Need" paper used $\beta_2 = 0.98$).

As discussed above, $\epsilon = 10^{-8}$ is the default. For float16 training, $\epsilon = 10^{-6}$ or $10^{-5}$ is often more stable.