Momentum › Choosing the Momentum Coefficient

The momentum coefficient $\beta$ is a hyperparameter you set before training. The typical values in practice and what they imply are:

• $\beta = 0.9$: The most common default. Provides a 10x effective learning rate amplification for persistent directions and reasonable oscillation damping. Works well for most problems.
• $\beta = 0.99$: Stronger momentum, 100x amplification. Useful for smoother loss landscapes or when you want very aggressive smoothing of noisy gradients. Requires a much smaller learning rate. The build-up to steady-state velocity takes about 100 steps, which can cause a noticeable initial slowdown.
• $\beta = 0.95$: A middle ground, often used in learning rate schedules that also ramp up momentum over the first few hundred steps.
• $\beta = 0$: Reduces to plain SGD, since the velocity equals the current gradient at every step.

The choice of $\beta$ interacts strongly with the learning rate schedule. If you increase $\beta$ during training (a technique used in some schedules), you must simultaneously reduce the learning rate to compensate for the increased effective rate. Forgetting to do this is a common source of instability when experimenting with momentum values.

One rule of thumb: if your training loss is oscillating wildly, you can either lower the learning rate or increase $\beta$ slightly to smooth out the trajectory. The two interventions have different effects: lowering the learning rate reduces the step size uniformly across all directions, while increasing $\beta$ specifically dampens high-frequency gradient components and amplifies low-frequency ones. When you have a specific diagnosis (oscillation indicates high-frequency gradient noise), increasing $\beta$ is the more targeted fix. When you are uncertain of the cause, reducing the learning rate is safer because it reduces the magnitude of all updates without changing their direction distribution.

In practice, $\beta = 0.9$ works across a wide range of architectures and learning rates. You can usually start with this default and only adjust it if you observe specific pathologies. The loss is more sensitive to the learning rate than to $\beta$, which is why learning rate tuning is prioritized in most hyperparameter search protocols. If you are constrained to only one tuning run, fix $\beta = 0.9$ and spend your budget on the learning rate.

Some problem types require an adjustment to the default. Very noisy gradient estimates (small batch sizes, highly stochastic problems) benefit from higher $\beta$ values (0.95 or above) to smooth the gradient signal. Problems with sharp, narrow minima (some natural language processing fine-tuning tasks) sometimes perform better with lower $\beta$ (0.85 or 0.8) to reduce the risk of overshooting. Recurrent network training, which can have highly variable gradient scales across time steps, sometimes benefits from careful $\beta$ tuning.