Momentum › Effective Learning Rate with Momentum

One important consequence of momentum is that it amplifies the effective learning rate for gradient components that are persistent across steps. To see why, consider the steady state where the gradient has been constant at $g$ for many steps, using the PyTorch formulation (without the $(1-\beta)$ factor).

Starting from $\mathbf{v}_0 = 0$ and applying the recursion $\mathbf{v}_t = \beta \, \mathbf{v}_{t-1} + g$ repeatedly:

$$\begin{aligned} \mathbf{v}_1 &= g \\ \mathbf{v}_2 &= \beta g + g = (1 + \beta) g \\ \mathbf{v}_t &= g \sum_{k=0}^{t-1} \beta^k \end{aligned}$$

As $t \to \infty$, the sum converges via the geometric series formula to:

$$\mathbf{v}_\infty = g \cdot \frac{1}{1 - \beta}$$

where:

• $g$: the constant gradient value in steady state
• $\beta$: the momentum coefficient
• $\mathbf{v}_\infty$: the limiting velocity after many steps of constant gradient

The actual step size applied to the parameters is $\alpha \cdot \mathbf{v}_\infty = \alpha \cdot g / (1 - \beta)$. Compared to plain SGD's step size of $\alpha \cdot g$, the effective learning rate is multiplied by $1/(1-\beta)$.

With $\beta = 0.9$, the effective rate for persistent directions is $\alpha / (1 - 0.9) = 10\alpha$. This means you should often use a smaller nominal learning rate with momentum than you would with plain SGD. With $\beta = 0.99$, the amplification is 100-fold, which requires an even smaller base learning rate to avoid divergence.

This scaling matters in practice: switching from plain SGD to SGD with $\beta = 0.9$ without adjusting the learning rate will often cause divergence, because the optimizer is now effectively taking 10x larger steps. The safe approach is to divide the learning rate by $(1-\beta)$ when adding momentum, keeping the effective step size constant. You can then separately tune the learning rate further.

The amplification also explains why momentum is so valuable when the loss surface has many flat regions. In a flat region, the gradient is small at every step. Plain SGD takes tiny steps and barely moves. With momentum, even small but consistent gradients accumulate over time, allowing the optimizer to build up meaningful velocity. This "free" speed-up in flat regions is a fundamental advantage of momentum over plain SGD that goes beyond just oscillation dampening. It is particularly important in modern deep learning, where loss surfaces often have long flat corridors connecting saddle points and the optimizer must traverse these corridors to reach good minima.