Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

AdamW › The Effect of Gradient History on Regularization Strength

Michael BrenndoerferApril 20, 20252 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 69AdamW › The Effect of Gradient History on Regularization Strength

Reading section 8 of 15 · The Effect of Gradient History on Regularization Strength · 2 minSection 8 of 15 · 2 min

Section 8 of 15 · 2 min read

AdamW › The Effect of Gradient History on Regularization StrengthOne section · Sections in this chapter8 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Another way to understand the difference is to look at how the effective regularization strength varies across parameters with different gradient histories. This is the analytical picture underlying the numerical example from earlier.

In[9]:

Code

# Simulate effective regularization strength across parameters
# with different gradient history magnitudes
np.random.seed(0)

# Imagine 1000 parameters with varying gradient history magnitudes
n_params = 1000
v_hat_values = np.logspace(-4, 0, n_params)  # sqrt(v_hat) from 0.01 to 1.0

# For a given weight decay lambda
lam = 0.01
epsilon = 1e-8

# Adam+L2: regularization passes through adaptive scaling
# Effective regularization = lam / (sqrt(v_hat) + eps)
effective_reg_adam_l2 = lam / (np.sqrt(v_hat_values) + epsilon)

# AdamW: regularization is independent of v_hat
effective_reg_adamw = np.full(n_params, lam)

Out[10]:

Visualization

Log-scale plot showing effective regularization strength versus gradient history magnitude for two optimizers. — Effective regularization strength per parameter as a function of gradient history magnitude (shown as the square root of the second moment estimate) for Adam+L2 versus AdamW. With Adam+L2, parameters with small historical gradients (left side) receive up to 100 times stronger effective regularization than parameters with large gradient histories (right side). AdamW's flat line at lambda=0.01 shows that all parameters receive exactly the same regularization strength, regardless of gradient history.

The plot makes the problem concrete. For parameters with small gradient histories (left side of the x-axis), Adam+L2 applies extremely strong effective regularization, up to 100 times stronger than for frequently-updated parameters (right side). AdamW's flat line demonstrates its key advantage: the regularization coefficient has a consistent, predictable meaning across all parameters. When you set weight_decay=0.01 in AdamW, every single parameter in the model is shrunk by a factor of $0.01 \times \alpha$ at each step, no more and no less.