Adam Optimizer › Adam Optimizer

Training a deep neural network is fundamentally an optimization problem: you have a loss function and you want to find the weights that minimize it. As we saw in the Stochastic Gradient Descent chapter, SGD moves weights in the direction of steepest descent, and momentum adds a velocity term to smooth out oscillations and speed past shallow gradients. Both methods use a single global learning rate for every parameter, which creates a fundamental tension. Some parameters receive dense gradient signals and should move cautiously; others are updated rarely and need larger steps to converge at all. A single learning rate cannot serve both needs well.

Adam (Adaptive Moment Estimation), introduced by Diederik Kingma and Jimmy Ba in 2014, resolves this tension by maintaining an independent learning rate for every parameter, scaled automatically based on the history of gradients that parameter has seen. It does this by tracking two running statistics: the mean of past gradients (the first moment) and the uncentered variance of past gradients (the second moment). These two statistics together give Adam enough information to set an appropriate step size for each parameter individually. The result is an optimizer that requires almost no per-problem tuning and converges reliably across a wide range of architectures, from convolutional networks to transformers.

Understanding Adam deeply means understanding why the two moment estimates are needed, where the bias correction terms come from, why the epsilon term in the denominator matters more than its small value suggests, and what the practical failure modes look like in production training runs. This chapter covers all of that, from first principles through a full PyTorch implementation, a survey of Adam's most important variants, and a discussion of the memory implications that make Adam challenging at large scale.