Momentum › Key Parameters

When configuring momentum in PyTorch's torch.optim.SGD, the key parameters are:

• momentum: The momentum coefficient $\beta$. Set to 0.9 as the default starting point. Increase toward 0.99 for smoother loss landscapes or noisier gradients; decrease if you observe unstable training.
• lr: The base learning rate. When using momentum with $\beta = 0.9$, start with a learning rate about 10x smaller than you would use for plain SGD, since the effective rate is amplified.
• nesterov: Set to True to use Nesterov momentum instead of classical Polyak momentum. Requires dampening=0. Often provides slightly faster convergence; rarely hurts.
• dampening: Scaling factor that reduces the gradient contribution to the velocity. Keep at 0 for standard momentum; values above 0 weaken the momentum effect and are rarely used.
• weight_decay: L2 regularization coefficient applied to the parameters, not the velocity. Separate from momentum and can be set independently.