Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

AdamW › AdamW vs. Adam: The Effect of Regularization Strength

Michael BrenndoerferApril 20, 20252 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 69AdamW › AdamW vs. Adam: The Effect of Regularization Strength

Reading section 10 of 15 · AdamW vs. Adam: The Effect of Regularization Strength · 2 minSection 10 of 15 · 2 min

Section 10 of 15 · 2 min read

AdamW › AdamW vs. Adam: The Effect of Regularization StrengthOne section · Sections in this chapter10 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The difference between Adam+L2 and AdamW is most visible when regularization matters most, specifically at larger regularization strengths and in models with heterogeneous gradient distributions. Let's run a systematic comparison across different weight decay values.

In[16]:

Code

# Compare final weight norms for different regularization strengths
reg_values = [0.0, 0.001, 0.01, 0.05, 0.1, 0.2]
final_norms_adam = []
final_norms_adamw = []

for reg in reg_values:
    _, norms_a, _ = train_with_norms(
        adam_l2_update, lr=1e-2, reg=reg, seed=42, use_weight_decay=False
    )
    _, norms_w, _ = train_with_norms(
        adamw_update, lr=1e-2, reg=reg, seed=42, use_weight_decay=True
    )
    final_norms_adam.append(norms_a[-1])
    final_norms_adamw.append(norms_w[-1])

Out[17]:

Visualization

Line plot comparing final weight norms at different regularization strengths for Adam L2 and AdamW. — Final weight norm at the end of 500 training epochs as a function of the regularization coefficient for Adam+L2 versus AdamW. As the regularization coefficient increases, AdamW produces more aggressively controlled weight norms that decrease predictably. Adam+L2 shows weaker and less consistent weight decay effects, particularly at higher regularization strengths, because the adaptive scaling of the second moment estimate partially cancels the intended decay for frequently-updated parameters.

The plot reveals how Adam+L2 and AdamW respond differently to increasing regularization. AdamW shows more consistent and predictable weight decay behavior: as you increase the regularization coefficient, the final weight norm decreases correspondingly. Adam+L2 shows a weaker response to increasing regularization, particularly at higher values, because the adaptive scaling is partially counteracting the intended effect of the L2 penalty for parameters with large gradient histories.