Momentum › The Momentum Update Equations

Momentum introduces a new variable, the velocity vector $\mathbf{v}_t$, which tracks an exponential moving average of past gradients. The update rule has two parts: first update the velocity, then update the parameters.

The exponential moving average of gradients accumulates directional information from past steps. Given a momentum coefficient $\beta \in [0, 1)$, the velocity is updated as:

$$\mathbf{v}_t = \beta \, \mathbf{v}_{t-1} + (1 - \beta) \, \nabla_\theta \mathcal{L}(\theta_{t-1})$$

where:

• $\mathbf{v}_t$: the velocity vector at step $t$, an exponential moving average of past gradients
• $\mathbf{v}_{t-1}$: the velocity from the previous step, initialized to $\mathbf{0}$ at $t=0$
• $\beta$: the momentum coefficient, controlling how strongly past velocity persists; typically 0.9
• $\nabla_\theta \mathcal{L}(\theta_{t-1})$: the gradient of the loss $\mathcal{L}$ with respect to parameters $\theta$ evaluated at the previous parameter values

Once the velocity is updated, the parameters move in the direction of the accumulated velocity:

$$\theta_t = \theta_{t-1} - \alpha \, \mathbf{v}_t$$

where:

• $\theta_t$: the updated parameter vector at step $t$
• $\theta_{t-1}$: the parameter vector before the update
• $\alpha$: the learning rate, a positive scalar controlling the magnitude of each update

The momentum coefficient $\beta$ determines the "half-life" of past gradients. A $\beta$ of 0.9 means that 90% of the previous velocity carries over to the next step, while the current gradient contributes 10%. A gradient from $k$ steps ago gets multiplied by $\beta^k$, so a gradient from 10 steps ago contributes with weight $0.9^{10} \approx 0.35$, meaning older gradients fade exponentially.

The intuition behind this two-variable structure is important. The velocity $\mathbf{v}_t$ plays the role of the ball's momentum in the physical analogy: it carries the history of where the ball has been going, not just where it is being pushed right now. The learning rate $\alpha$ scales how much the accumulated velocity translates into actual parameter movement. These two roles are cleanly separated, which makes it easy to reason about their effects independently.

The velocity vector has a direct geometric interpretation. At any point in training, $\mathbf{v}_t$ is a vector in the same space as the parameters: it has one entry per parameter. If the network has 10 million parameters, the velocity vector has 10 million entries, each tracking the exponentially smoothed gradient for that specific parameter. The velocity for one parameter does not interact with the velocity for another; they evolve independently under the same $\beta$ schedule. This parameter-wise independence is a feature, not a limitation: it means momentum scales to any network size without additional complexity, and it means that the direction-alignment property works out properly even in high-dimensional spaces.

You will often see momentum written in a slightly different form, sometimes called the "heavy ball" formulation or Polyak momentum:

$$\mathbf{v}_t = \beta \, \mathbf{v}_{t-1} + \nabla_\theta \mathcal{L}(\theta_{t-1})$$ $$\theta_t = \theta_{t-1} - \alpha \, \mathbf{v}_t$$

Here the gradient is added without the $(1 - \beta)$ scaling factor. This is the formulation used in PyTorch's SGD optimizer by default. The two forms are mathematically equivalent up to a rescaling of the learning rate. With the $(1-\beta)$ form, the accumulated velocity in steady state equals the gradient itself, while the PyTorch form accumulates a scaled-up velocity. To get the same effective step size, you compensate by adjusting $\alpha$ accordingly.

The reason PyTorch uses the formulation without the $(1-\beta)$ factor is largely historical convention: it matches the original formulation proposed by Boris Polyak in 1964 and the form used in most early deep learning frameworks. The $(1-\beta)$ normalized form is more common in the statistical literature on exponential smoothing. Both are correct; choose whichever formulation matches the software you are using, and be careful when comparing hyperparameter values between implementations that use different conventions.

One of momentum's great practical advantages is its minimal overhead. Each parameter requires exactly one additional number (the velocity) to be stored and updated. For a network with $P$ parameters, momentum adds $P$ additional floats to the optimizer's memory footprint. At 32-bit precision, this is $4P$ extra bytes. For a model with 100 million parameters, this is 400 MB of additional memory: substantial, but the same order as storing the parameters themselves.

The computation cost is also minimal. The velocity update involves one scalar multiplication ($\beta \cdot \mathbf{v}_{t-1}$), one scalar multiplication or addition (the gradient term), and one vector addition. These operations are performed in parallel across all parameters via vectorized hardware instructions and have negligible runtime compared to the forward pass and backpropagation. In practice, the computational overhead of momentum is unmeasurable on modern hardware.

This cost profile is why momentum has persisted as a fundamental component even as more sophisticated adaptive methods have emerged. Adding momentum to SGD costs nothing at inference time (the velocity is only needed during training) and next to nothing during training itself.