Momentum › Key Parameters

Michael BrenndoerferApril 23, 20251 min read
Language AI HandbookAll chapters
Part 10 of 67Neural Network Foundations
Chapter 67Momentum › Key Parameters
Reading section 13 of 18 · Key Parameters · 1 minSection 13 of 18 · 1 min

Section 13 of 18 · 1 min read

Momentum › Key ParametersSection view · Sections in this chapter13 / 18 ↓
Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

When configuring momentum in PyTorch's torch.optim.SGD, the key parameters are:

  • momentum: The momentum coefficient β\beta. Set to 0.9 as the default starting point. Increase toward 0.99 for smoother loss landscapes or noisier gradients; decrease if you observe unstable training.
  • lr: The base learning rate. When using momentum with β=0.9\beta = 0.9, start with a learning rate about 10x smaller than you would use for plain SGD, since the effective rate is amplified.
  • nesterov: Set to True to use Nesterov momentum instead of classical Polyak momentum. Requires dampening=0. Often provides slightly faster convergence; rarely hurts.
  • dampening: Scaling factor that reduces the gradient contribution to the velocity. Keep at 0 for standard momentum; values above 0 weaken the momentum effect and are rarely used.
  • weight_decay: L2 regularization coefficient applied to the parameters, not the velocity. Separate from momentum and can be set independently.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Newsletter

Stay up to date

Get new articles, book updates, and notes on AI and technology.

Unsubscribe at any time. Privacy policy