Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Adam Optimizer › Visualizing Adam's Adaptive Behavior

Michael BrenndoerferApril 24, 20257 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 68Adam Optimizer › Visualizing Adam's Adaptive Behavior

Reading section 13 of 18 · Visualizing Adam's Adaptive Behavior · 7 minSection 13 of 18 · 7 min

Section 13 of 18 · 7 min read

Adam Optimizer › Visualizing Adam's Adaptive BehaviorOne section · Sections in this chapter13 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To understand how Adam adapts to different gradient patterns, let's build visualizations that trace the moment estimates and effective learning rates for parameters with different gradient histories.

Out[15]:

Visualization

Line plot showing gradient and first moment over 50 steps, first moment smoothing gradient noise. — First moment estimate over 50 steps for a parameter receiving consistent gradients with noise. The bias-corrected first moment smooths the noisy raw gradient, converging toward the true mean of 0.3. Individual gradient samples vary widely due to noise, while the moment estimate remains well-behaved throughout training.

Line plot showing second moment trajectories for three parameters with different gradient magnitudes. — Bias-corrected second moment estimates over 50 steps for three parameters with different gradient magnitudes. Parameters with larger gradients accumulate larger second-moment values, which Adam uses to compress their effective learning rates. The high-magnitude parameter stabilizes quickly; the low-magnitude parameter climbs slowly.

Out[16]:

Visualization

Bar chart showing effective learning rates for three parameter types: sparse, medium, and dense gradient history. — Effective learning rate per parameter after 200 training steps for three gradient-density patterns. Sparse parameters (5% chance of receiving a gradient) end up with a much larger effective learning rate than dense parameters (90% chance), because their smaller accumulated second moment allows larger steps. The dashed line shows the global learning rate alpha = 0.001 for reference.

The visualization confirms the core intuition: parameters that receive large, frequent gradients get a smaller effective learning rate, while parameters that receive rare or small gradients get a larger boost. The global learning rate $\alpha = 0.001$ is a reference, with the adaptive normalization adjusting each parameter up or down based on its gradient history.

Out[17]:

Visualization

Log-scale line plot comparing Adam and SGD training loss over 300 steps. — Training loss over 300 steps for Adam and SGD with momentum on an ill-conditioned quadratic surface. Adam converges faster because its adaptive learning rates compensate for the 10x curvature difference between dimensions. SGD must use a small global learning rate to avoid diverging in the steep dimension, slowing convergence in the flat dimension.

Contour plot with optimization paths for Adam and SGD on an ill-conditioned quadratic. — Optimization paths on the 2D loss surface. SGD with momentum oscillates across the narrow valley (steep x-direction, flat y-direction) while Adam navigates more directly toward the minimum, making efficient use of the different curvatures along each axis.

The ill-conditioned quadratic (where the curvature in the $x$ -direction is 10x steeper than in $y$ ) illustrates where adaptive learning rates shine. SGD with momentum oscillates across the narrow valley because it must use a conservative learning rate to avoid overshooting in the steep $x$ -dimension. Adam's adaptive normalization effectively uses a different learning rate along each dimension, letting it move along the valley directly. The same geometry appears in transformer loss surfaces: some parameter directions are far stiffer than others, which is one important reason Adam dominates transformer training.

Out[18]:

Visualization

Line plot showing learning rate over 1000 steps with linear warmup and cosine decay. — Learning rate schedule with linear warmup (0 to 100 steps) followed by cosine decay (100 to 1000 steps). The warmup phase prevents unstable early updates by giving Adam time to build reliable moment estimates before large steps are taken. The cosine decay then gradually reduces the learning rate, allowing fine-grained convergence in later training and preventing the optimizer from bouncing around near the minimum.