Momentum: SGD Optimization with Velocity Accumulation

Imagine rolling a ball down a hilly landscape. The ball does not restart from zero speed at every step. It carries the velocity it has built up, accelerating through valleys and slowing on uphill slopes. That physical intuition describes exactly what momentum does for neural network optimization. Where vanilla stochastic gradient descent (SGD) moves by a fixed-size step in the direction of the steepest descent at each iteration, momentum lets the optimizer accumulate a sense of direction and speed across iterations, so it can race through flat regions and sail over small bumps without getting stuck.

We covered SGD in the previous chapter, where we saw that the learning rate controls how far we move on each step. But even a well-tuned learning rate struggles in one common scenario: loss surfaces that are much steeper in one dimension than another. In such narrow valley landscapes, plain SGD oscillates side to side across the narrow dimension while making frustratingly slow progress along the valley floor toward the minimum. Momentum solves this by remembering past gradients and letting the optimizer naturally dampen oscillations while building up speed in persistent directions.

This chapter covers the mathematics of momentum, the intuition behind it, Polyak (classical) momentum versus Nesterov momentum, the effective learning rate phenomenon, how momentum interacts with learning rate schedules, and how to configure momentum in PyTorch. By the end, you will understand why momentum is rarely disabled in modern deep learning training runs, how to choose its hyperparameters in practice, and where its limitations lie relative to adaptive methods like Adam.

Before diving into the mechanics, it helps to understand the problem momentum is designed to solve. The shortcomings of plain SGD are not immediately obvious if you only consider loss surfaces shaped like a round bowl. On a spherically symmetric loss surface, SGD performs perfectly well: every gradient step points directly toward the minimum, and with the right learning rate, you converge efficiently. The problem surfaces when the loss curvature is highly non-isotropic, meaning it differs dramatically across different directions in parameter space. This distinction between easy and hard loss landscapes is needed to understand why virtually every modern training run uses some form of momentum.

Consider a quadratic bowl whose curvature is much stronger in one direction than the other: a coin-shaped bowl where east-west is very steep but north-south is almost flat. This kind of landscape is extremely common in deep learning. Neural networks with many layers and skip connections often have loss surfaces that look like narrow corridors along many axes simultaneously. Even a simple two-layer network can exhibit curvature ratios of 100:1 or higher between its fastest and slowest-changing weight directions.

Plain SGD takes a step in whatever direction the current gradient points. At a point on the steep sidewall, the gradient is dominated by the steep direction. The step overshoots the valley center, lands on the opposite wall, overshoots back, and so on. The optimizer zigzags laterally while barely progressing along the valley's gentle floor toward the minimum.

This zigzagging behavior is not just inefficient. It actively prevents the use of larger learning rates, because a larger step size makes the sideways oscillation worse. You are forced to use a small learning rate to control lateral noise, which then makes progress along the shallow floor glacially slow. You are essentially caught in a tradeoff: a learning rate large enough to make progress along the gentle axis is too large to avoid oscillation along the steep axis.

To visualize how extreme this can get: if the curvature in the steep direction is 100 times the curvature in the shallow direction, an SGD step that moves one unit along the shallow direction moves 100 units across the steep direction, instantly overshooting the valley center. To keep the lateral step small enough to not overshoot, you must use a learning rate 100 times smaller than you would otherwise use, which makes every step along the shallow direction only 1/100 of a unit. Progress is agonizingly slow.

The fundamental cause of this problem is that a single learning rate must serve all parameter directions simultaneously. It needs to be small enough for the steepest direction and large enough for the shallowest direction, and these two requirements conflict when the curvature ratio is large. SGD has no mechanism to allocate different effective step sizes to different parameter directions, so it is always limited by the worst-case direction.

The ideal behavior is to accumulate momentum in the direction of consistent gradient information (the floor direction) while cancelling out gradient components that flip sign repeatedly (the wall direction). A gradient that pointed left last step and now points right contributes nothing in the accumulated direction, but a gradient that pointed forward both last step and this step compounds, making the optimizer move faster along the consistent axis.

This property emerges naturally from exponential averaging of gradients. If you take a weighted average of many past gradient vectors, the components that consistently agree in sign survive the averaging and amplify each other. The components that alternate sign tend to cancel out. The result is a filtered representation of the gradient that suppresses noise and oscillations while preserving the signal.

That is exactly what momentum achieves. It acts as a low-pass filter on the gradient signal, removing high-frequency oscillations while preserving the low-frequency trend toward the minimum. This filtering perspective is mathematically precise: if you analyze the gradient signal as a time series, momentum is a single-pole infinite impulse response (IIR) low-pass filter with a cutoff frequency determined by the momentum coefficient $\beta$. Higher $\beta$ means more aggressive filtering, cutting out more high-frequency noise but also introducing more lag between a change in the true gradient direction and the optimizer's response.

There is a second pathological landscape that motivates momentum beyond narrow valleys: saddle points. A saddle point is a location where the gradient is zero, but the point is neither a local minimum nor a local maximum. At a saddle point, the loss curves upward in some directions and downward in others.

Saddle points are believed to be far more common than local minima in high-dimensional neural network loss surfaces. The argument comes from random matrix theory: for a critical point to be a local minimum, all eigenvalues of the Hessian must be positive. In a 1,000-dimensional parameter space, the probability that all 1,000 eigenvalues are simultaneously positive approaches zero for generic random functions, while the probability of a saddle point (some positive, some negative eigenvalues) is high.

At a saddle point, plain SGD stalls because the gradient is zero and there is no signal to move. In practice, the gradient is never exactly zero due to floating-point noise and minibatch stochasticity, so SGD does not completely freeze. But it slows dramatically near saddle points, sometimes appearing stuck for hundreds of iterations before escaping. Momentum, by contrast, carries the optimizer through the saddle on the strength of the velocity it has built up approaching the saddle. The ball analogy applies here too: a ball rolling along a flat table does not stop when the slope becomes zero; it coasts forward on its inertia.

This saddle-escaping property is one reason researchers have repeatedly found that momentum significantly reduces training time even on smooth, well-conditioned problems, not just on the narrow valley problems where the oscillation-damping explanation applies most directly.

Momentum introduces a new variable, the velocity vector $\mathbf{v}_t$, which tracks an exponential moving average of past gradients. The update rule has two parts: first update the velocity, then update the parameters.

The exponential moving average of gradients accumulates directional information from past steps. Given a momentum coefficient $\beta \in [0, 1)$, the velocity is updated as:

where:

• $\mathbf{v}_t$: the velocity vector at step $t$, an exponential moving average of past gradients
• $\mathbf{v}_{t-1}$: the velocity from the previous step, initialized to $\mathbf{0}$ at $t=0$
• $\beta$: the momentum coefficient, controlling how strongly past velocity persists; typically 0.9
• $\nabla_\theta \mathcal{L}(\theta_{t-1})$: the gradient of the loss $\mathcal{L}$ with respect to parameters $\theta$ evaluated at the previous parameter values

Once the velocity is updated, the parameters move in the direction of the accumulated velocity:

where:

• $\theta_t$: the updated parameter vector at step $t$
• $\theta_{t-1}$: the parameter vector before the update
• $\alpha$: the learning rate, a positive scalar controlling the magnitude of each update

The momentum coefficient $\beta$ determines the "half-life" of past gradients. A $\beta$ of 0.9 means that 90% of the previous velocity carries over to the next step, while the current gradient contributes 10%. A gradient from $k$ steps ago gets multiplied by $\beta^k$, so a gradient from 10 steps ago contributes with weight $0.9^{10} \approx 0.35$, meaning older gradients fade exponentially.

The intuition behind this two-variable structure is important. The velocity $\mathbf{v}_t$ plays the role of the ball's momentum in the physical analogy: it carries the history of where the ball has been going, not just where it is being pushed right now. The learning rate $\alpha$ scales how much the accumulated velocity translates into actual parameter movement. These two roles are cleanly separated, which makes it easy to reason about their effects independently.

The velocity vector has a direct geometric interpretation. At any point in training, $\mathbf{v}_t$ is a vector in the same space as the parameters: it has one entry per parameter. If the network has 10 million parameters, the velocity vector has 10 million entries, each tracking the exponentially smoothed gradient for that specific parameter. The velocity for one parameter does not interact with the velocity for another; they evolve independently under the same $\beta$ schedule. This parameter-wise independence is a feature, not a limitation: it means momentum scales to any network size without additional complexity, and it means that the direction-alignment property works out properly even in high-dimensional spaces.

You will often see momentum written in a slightly different form, sometimes called the "heavy ball" formulation or Polyak momentum:

Here the gradient is added without the $(1 - \beta)$ scaling factor. This is the formulation used in PyTorch's SGD optimizer by default. The two forms are mathematically equivalent up to a rescaling of the learning rate. With the $(1-\beta)$ form, the accumulated velocity in steady state equals the gradient itself, while the PyTorch form accumulates a scaled-up velocity. To get the same effective step size, you compensate by adjusting $\alpha$ accordingly.

The reason PyTorch uses the formulation without the $(1-\beta)$ factor is largely historical convention: it matches the original formulation proposed by Boris Polyak in 1964 and the form used in most early deep learning frameworks. The $(1-\beta)$ normalized form is more common in the statistical literature on exponential smoothing. Both are correct; choose whichever formulation matches the software you are using, and be careful when comparing hyperparameter values between implementations that use different conventions.

One of momentum's great practical advantages is its minimal overhead. Each parameter requires exactly one additional number (the velocity) to be stored and updated. For a network with $P$ parameters, momentum adds $P$ additional floats to the optimizer's memory footprint. At 32-bit precision, this is $4P$ extra bytes. For a model with 100 million parameters, this is 400 MB of additional memory: substantial, but the same order as storing the parameters themselves.

The computation cost is also minimal. The velocity update involves one scalar multiplication ($\beta \cdot \mathbf{v}_{t-1}$), one scalar multiplication or addition (the gradient term), and one vector addition. These operations are performed in parallel across all parameters via vectorized hardware instructions and have negligible runtime compared to the forward pass and backpropagation. In practice, the computational overhead of momentum is unmeasurable on modern hardware.

This cost profile is why momentum has persisted as a fundamental component even as more sophisticated adaptive methods have emerged. Adding momentum to SGD costs nothing at inference time (the velocity is only needed during training) and next to nothing during training itself.

The classical analogy for momentum is a ball rolling down a hilly landscape under gravity and friction. The gradient of the loss function corresponds to the slope the ball experiences at its current position. The velocity of the ball is the momentum variable $\mathbf{v}_t$, and friction is controlled by $(1 - \beta)$.

When the ball rolls into a narrow valley, it naturally builds up speed along the floor because the floor gradient consistently points in one direction. The oscillations perpendicular to the floor cancel themselves out over time because they alternate direction. The ball effectively averages out the lateral oscillations and accelerates down the valley.

The coefficient $\beta$ plays the role of friction. At $\beta = 0$, there is maximum friction; the ball's velocity is reset to the current gradient at every step, which is identical to plain SGD. At $\beta \to 1$, friction disappears; the ball accumulates speed indefinitely and overshoots dramatically. The practical sweet spot is $\beta = 0.9$, which provides enough persistence to dampen oscillations while preventing the ball from running so far that it overshoots the minimum.

This analogy predicts behaviors you will observe in training. When you first start training, the ball is stationary and it takes a few steps to build up speed, just as we observe that momentum optimizers have a slow start before accelerating. When training approaches a minimum, the ball overshoots if it has built up too much speed, which matches the observed tendency of high-$\beta$ momentum to oscillate around minima. When you hit a flat saddle region, a ball already moving does not stop at the saddle, it coasts through; this matches the empirical observation that momentum helps escape saddle points faster than plain SGD.

The analogy also helps you understand what happens in high dimensions. Imagine not a two-dimensional bowl but a 10-million-dimensional loss surface. Each parameter direction has its own independent "slope" and the ball rolls simultaneously in all 10 million directions. In some directions the slope is steep and consistent (carrying the ball quickly toward the minimum along those axes), in some directions it alternates (cancelled out), and in some directions it is nearly flat (the ball barely moves). Momentum's filtering action applies simultaneously in all directions, independently, and the aggregate effect is that the optimizer allocates its effective learning rate budget preferentially to the directions where the gradient signal is most consistent.

To make the oscillation-dampening effect concrete, consider what happens to gradient components in a narrow valley.

Along the valley floor (shallow direction), the gradient consistently points toward the minimum. Each step adds a positive contribution to $\mathbf{v}_t$ in this direction. The velocity in the floor direction accumulates, so the effective step size in this direction grows over time.

Across the valley (steep direction), the gradient alternates sign every step: first positive (pointing left), then negative (pointing right), then positive again. With momentum, these alternating contributions partially cancel in the exponential moving average. If the gradient alternates as $+g, -g, +g, -g, \ldots$, consecutive terms cancel and the velocity in that direction approaches zero.

The net effect is that the optimizer moves quickly toward the minimum along the floor while barely oscillating across the steep walls. More precisely, if a gradient component is consistently $+g$ across many steps, the velocity in that direction builds toward $g/(1-\beta)$ in the PyTorch formulation. If it alternates sign, the contributions cancel and velocity stays near zero.

This asymmetry is the heart of why momentum is effective. The optimizer is not just moving faster: it is moving smarter, allocating effective learning rate capacity to directions where the gradient provides consistent, trustworthy information, and withholding it from directions where the gradient is noisy or oscillatory.

Let us think about this more carefully using the frequency domain analogy. Each parameter direction's gradient, viewed as a function of training step, is a signal. A gradient component that consistently points in one direction is a low-frequency signal (constant, slowly varying). A gradient component that oscillates back and forth every step is a high-frequency signal (alternating). The exponential moving average with coefficient $\beta$ attenuates high-frequency signals and passes low-frequency signals, just like an electronic low-pass filter. The cutoff frequency is approximately $(1-\beta)/(2\pi)$ in normalized units. Lower $\beta$ means higher cutoff frequency, meaning less filtering. Higher $\beta$ means lower cutoff frequency, meaning more aggressive smoothing.

This frequency interpretation reveals something important: momentum is not eliminating noise by averaging multiple samples of the same quantity (as you would in a batch size increase). It is eliminating directional noise by distinguishing between gradient directions that are persistent (signal) and gradient directions that fluctuate (noise). A single sample per step is enough for momentum to work because it exploits temporal correlation across steps rather than spatial averaging across samples.

One important consequence of momentum is that it amplifies the effective learning rate for gradient components that are persistent across steps. To see why, consider the steady state where the gradient has been constant at $g$ for many steps, using the PyTorch formulation (without the $(1-\beta)$ factor).

Starting from $\mathbf{v}_0 = 0$ and applying the recursion $\mathbf{v}_t = \beta \, \mathbf{v}_{t-1} + g$ repeatedly:

As $t \to \infty$, the sum converges via the geometric series formula to:

where:

• $g$: the constant gradient value in steady state
• $\beta$: the momentum coefficient
• $\mathbf{v}_\infty$: the limiting velocity after many steps of constant gradient

The actual step size applied to the parameters is $\alpha \cdot \mathbf{v}_\infty = \alpha \cdot g / (1 - \beta)$. Compared to plain SGD's step size of $\alpha \cdot g$, the effective learning rate is multiplied by $1/(1-\beta)$.

With $\beta = 0.9$, the effective rate for persistent directions is $\alpha / (1 - 0.9) = 10\alpha$. This means you should often use a smaller nominal learning rate with momentum than you would with plain SGD. With $\beta = 0.99$, the amplification is 100-fold, which requires an even smaller base learning rate to avoid divergence.

This scaling matters in practice: switching from plain SGD to SGD with $\beta = 0.9$ without adjusting the learning rate will often cause divergence, because the optimizer is now effectively taking 10x larger steps. The safe approach is to divide the learning rate by $(1-\beta)$ when adding momentum, keeping the effective step size constant. You can then separately tune the learning rate further.

The amplification also explains why momentum is so valuable when the loss surface has many flat regions. In a flat region, the gradient is small at every step. Plain SGD takes tiny steps and barely moves. With momentum, even small but consistent gradients accumulate over time, allowing the optimizer to build up meaningful velocity. This "free" speed-up in flat regions is a fundamental advantage of momentum over plain SGD that goes beyond just oscillation dampening. It is particularly important in modern deep learning, where loss surfaces often have long flat corridors connecting saddle points and the optimizer must traverse these corridors to reach good minima.

The momentum coefficient $\beta$ is a hyperparameter you set before training. The typical values in practice and what they imply are:

• $\beta = 0.9$: The most common default. Provides a 10x effective learning rate amplification for persistent directions and reasonable oscillation damping. Works well for most problems.
• $\beta = 0.99$: Stronger momentum, 100x amplification. Useful for smoother loss landscapes or when you want very aggressive smoothing of noisy gradients. Requires a much smaller learning rate. The build-up to steady-state velocity takes about 100 steps, which can cause a noticeable initial slowdown.
• $\beta = 0.95$: A middle ground, often used in learning rate schedules that also ramp up momentum over the first few hundred steps.
• $\beta = 0$: Reduces to plain SGD, since the velocity equals the current gradient at every step.

The choice of $\beta$ interacts strongly with the learning rate schedule. If you increase $\beta$ during training (a technique used in some schedules), you must simultaneously reduce the learning rate to compensate for the increased effective rate. Forgetting to do this is a common source of instability when experimenting with momentum values.

One rule of thumb: if your training loss is oscillating wildly, you can either lower the learning rate or increase $\beta$ slightly to smooth out the trajectory. The two interventions have different effects: lowering the learning rate reduces the step size uniformly across all directions, while increasing $\beta$ specifically dampens high-frequency gradient components and amplifies low-frequency ones. When you have a specific diagnosis (oscillation indicates high-frequency gradient noise), increasing $\beta$ is the more targeted fix. When you are uncertain of the cause, reducing the learning rate is safer because it reduces the magnitude of all updates without changing their direction distribution.

In practice, $\beta = 0.9$ works across a wide range of architectures and learning rates. You can usually start with this default and only adjust it if you observe specific pathologies. The loss is more sensitive to the learning rate than to $\beta$, which is why learning rate tuning is prioritized in most hyperparameter search protocols. If you are constrained to only one tuning run, fix $\beta = 0.9$ and spend your budget on the learning rate.

Some problem types require an adjustment to the default. Very noisy gradient estimates (small batch sizes, highly stochastic problems) benefit from higher $\beta$ values (0.95 or above) to smooth the gradient signal. Problems with sharp, narrow minima (some natural language processing fine-tuning tasks) sometimes perform better with lower $\beta$ (0.85 or 0.8) to reduce the risk of overshooting. Recurrent network training, which can have highly variable gradient scales across time steps, sometimes benefits from careful $\beta$ tuning.

Yurii Nesterov proposed a modification to classical momentum that generally converges faster, known as Nesterov Accelerated Gradient (NAG) or Nesterov momentum. The insight behind it is subtle but powerful: rather than computing the gradient at where you currently are, compute it at where you are about to go.

Classical Polyak momentum computes the gradient at the current position $\theta_{t-1}$, then applies the accumulated velocity. But if you know you are going to move in the direction of the velocity $\mathbf{v}_{t-1}$ anyway, why not compute the gradient at the position you will be after applying the velocity, rather than where you currently are?

This lookahead idea is more powerful than it first appears. In classical momentum, the gradient is "stale": you compute it at your current position, but by the time the gradient information is used (after applying momentum), you have already moved. The gradient may be pointing in a slightly wrong direction because it does not account for where you are heading. Nesterov fixes this by peering ahead.

Nesterov momentum defines a lookahead position by applying the committed momentum step:

where:

• $\theta_{\text{look}}$: the lookahead position where the gradient is evaluated
• $\alpha \, \beta \, \mathbf{v}_{t-1}$: the portion of the momentum step already committed before the current gradient correction

Then the update uses the gradient evaluated at this lookahead position:

The key difference from classical momentum is that Nesterov evaluates the gradient at the position after the momentum step, so the gradient acts as a correction to the committed velocity rather than being evaluated at the current (pre-step) position.

To extend the ball-rolling analogy: in classical momentum, the ball looks at the slope where it currently is and decides how to adjust. In Nesterov momentum, the ball first commits to rolling forward with its current velocity, then looks at the slope it will be on after rolling, and adjusts course based on what it sees there. The second ball gets a more accurate picture of where it is going, and can brake or steer more appropriately before arriving.

A concrete scenario illustrates why this matters. Suppose the optimizer is approaching a minimum at the bottom of a valley. Its accumulated velocity is pointing toward the minimum and is quite large from building up over many steps. In classical momentum, the gradient is evaluated at the current (pre-step) position, which still points generally toward the minimum, so the velocity update does not reduce the velocity enough to prevent overshooting. In Nesterov momentum, the gradient is evaluated at the position after applying the velocity, which may already be past the minimum. There, the gradient points back away from the minimum. This provides a corrective signal that slows down the velocity before the parameter update is applied. The result is less overshoot.

For convex problems, Nesterov's method achieves an $O(1/t^2)$ convergence rate compared to $O(1/t)$ for classical gradient descent without momentum. This theoretical result was a breakthrough in optimization theory when Nesterov proved it in 1983. The $O(1/t^2)$ rate is optimal for first-order methods on convex functions, meaning you cannot do better without accessing second-order information (like the Hessian). This optimality result is one of the most celebrated in mathematical optimization and placed Nesterov's method in the canon of provably optimal algorithms.

The intuition is that the lookahead gradient provides more informative updates. If the momentum is carrying the optimizer toward a bump or local increase in loss, the lookahead gradient will detect this earlier and apply a corrective update before overshooting. Classical momentum does not get this early warning and may overshoot before the gradient at the post-step position signals the problem.

In practice, Nesterov momentum often converges slightly faster than classical momentum on neural network problems, though the gap is typically modest for well-tuned hyperparameters. The theoretical $O(1/t^2)$ advantage applies to convex problems, and neural network loss surfaces are highly non-convex. Nevertheless, the practical advantage is consistently positive across a wide range of experiments, and since it costs nothing to use Nesterov (same computation), it is generally recommended as the default choice.

The gap between Nesterov and classical momentum tends to be more visible in two regimes: early in training when the optimizer is moving quickly through a changing loss landscape (where the stale gradient problem is most acute), and near the end of training when fine-grained convergence matters (where the corrective lookahead signal helps avoid overshooting). In the middle of training when the optimizer is making large, consistent progress toward a basin, both methods behave similarly.

Computing $\nabla_\theta \mathcal{L}(\theta_{\text{look}})$ at first appears to require evaluating the loss at a separate point, which would double the computational cost. Fortunately, there is an algebraically equivalent reformulation in terms of a reparameterized variable $\phi_t = \theta_t - \alpha \, \beta \, \mathbf{v}_t$ (the lookahead parameters at step $t$). The update can be written entirely in terms of $\phi$ with the gradient evaluated at the current $\phi_t$, making the implementation as cheap as classical momentum with just one gradient evaluation per step. PyTorch uses this reformulation internally when nesterov=True is set, which is why there is no computational overhead to enabling Nesterov.

To understand why this reformulation works, consider that the lookahead position at step $t$ is just the expected position of the parameters after applying the velocity at step $t$. By reparameterizing the optimization to track these lookahead positions rather than the original positions, you can express the entire algorithm in terms of quantities that are available at each step without any additional forward passes. This algebraic sleight-of-hand is what makes Nesterov's method practically viable: the theoretical elegance of the lookahead idea does not come with any implementation penalty.

PyTorch's torch.optim.SGD supports both classical and Nesterov momentum through the momentum and nesterov parameters.

The dampening parameter is a lesser-known option that scales down the gradient contribution to the velocity update. With dampening=d, the update becomes:

Setting dampening=0 gives the standard PyTorch momentum formulation. Nesterov momentum requires dampening=0. The dampening parameter is rarely tuned in practice and should be left at its default of 0 for standard momentum training.

Using momentum in a training loop is identical to using plain SGD; the optimizer handles the velocity state internally. PyTorch stores the velocity buffer as a stateful parameter in the optimizer, so it persists correctly across batches and epochs without any manual management.

The training loop is straightforward: zero_grad() clears old gradients, backward() computes new gradients, and step() applies the momentum-based update. The optimizer maintains the velocity buffer internally, so you do not need to track it manually. Notice that the interface is identical to plain SGD; momentum is configured once at optimizer creation and then operates transparently during training.

The zero_grad() call clears the accumulated parameter gradients but does not touch the velocity buffer. The velocity persists across iterations because that is the whole point: it accumulates information from previous steps. If you want to reset the momentum, you must either re-create the optimizer or manually clear the state_dict entries corresponding to the momentum buffers.

To make the difference concrete, let us train the same network on the same data with three optimizers and compare their training loss curves.

Both momentum variants achieve largely lower loss than plain SGD within the same number of epochs. This shows the convergence acceleration. The speedup is meaningful: momentum changes how effectively the optimizer uses each gradient computation rather than providing only a marginal improvement.

The effective learning rate curve shows why momentum choice matters so much. At $\beta = 0.9$ (the default), the effective step size for persistent gradient directions is 10 times the nominal learning rate. This is why practitioners typically reduce the base learning rate when switching from plain SGD to SGD with momentum. The relationship between $\beta$ and the multiplier is nonlinear and accelerates rapidly as $\beta$ approaches 1, which is why the difference between $\beta = 0.95$ and $\beta = 0.99$ is so dramatic in practice.

Momentum does not exist in isolation; it interacts in important ways with learning rate scheduling. Understanding these interactions is critical for getting the most out of momentum in real training workflows, and ignoring them is a common source of subtle training instability.

At the start of training, the velocity $\mathbf{v}_t$ is initialized to zero. The effective learning rate starts low and gradually builds up as the velocity accumulates. This natural warm-up effect means that the first few steps of training are more conservative, which can be beneficial for stability. Some practitioners rely on this behavior and do not add an explicit warm-up schedule when using momentum.

The time constant for velocity build-up is roughly $1/(1-\beta)$ steps. With $\beta = 0.9$, it takes about 10 steps for the velocity to reach approximately 63% of its steady-state value. With $\beta = 0.99$, the build-up takes about 100 steps. For very large $\beta$ values, this implicit warm-up becomes so long that it may interact badly with batch scheduling or other training decisions made early in the run.

This warm-up property also means that momentum has a self-correcting behavior when the gradient changes direction significantly. After a large direction change (for example, entering a new basin of the loss landscape), the momentum from the previous direction gradually dies out while the new direction builds up. The time constant is the same $1/(1-\beta)$ steps. This makes momentum naturally adaptive to changes in the loss landscape structure, at the cost of some lag in response.

Some learning rate schedules, such as the 1cycle policy popularized by Leslie Smith, ramp up the momentum at the start of training and then ramp it down, while doing the opposite with the learning rate. The intuition is that early in training, you want the optimizer to explore broadly (lower momentum, higher learning rate), and later you want it to zoom in precisely (higher momentum, lower learning rate).

This counterintuitive pairing (momentum up as learning rate down) can achieve very fast convergence and is used in fast.ai's training recipes. The key insight is that the effective step size is proportional to $\alpha / (1 - \beta)$, so you can keep the effective step size roughly constant while increasing $\beta$ by proportionally decreasing $\alpha$. What changes is the filtering quality: at the end of training, the optimizer is using very smooth gradient estimates (high $\beta$) with a smaller base learning rate, allowing it to find precise minima without the noisiness of early training.

The 1cycle approach has been validated empirically across many architectures and datasets. It achieves state-of-the-art results faster than fixed-schedule training in many benchmarks, largely because of this momentum-learning rate pairing. If you are looking for a training recipe to try beyond the standard fixed $\beta = 0.9$, the 1cycle policy with momentum scheduling is a well-validated option.

When using cosine annealing with warm restarts (a common schedule for modern training), the velocity should be reset to zero at each restart along with the learning rate reset. If you do not reset the velocity, the accumulated momentum from the previous cycle will carry the optimizer in a direction that may no longer be appropriate after the restart, undermining the intended fresh exploration.

In PyTorch, you can reset the velocity buffer by re-creating the optimizer or by manually clearing the momentum buffers from the optimizer's state dict. Re-creating the optimizer is simpler and less error-prone:

Forgetting to reset the momentum at restarts is a subtle bug that can be hard to diagnose because the training loss may still decrease; it just decreases more slowly than expected, and the benefits of warm restarts are reduced. If you are using warm restarts and not seeing the characteristic loss dips at each restart, forgetting to reset the velocity is one possible cause.

In large-scale training where you accumulate gradients over multiple mini-batches before each optimizer step (to simulate a larger effective batch size), the interaction with momentum requires care. The accumulated gradient represents the average gradient over multiple mini-batches, which is typically a lower-variance estimate than a single mini-batch gradient. This means the momentum coefficient can often be set slightly lower (less smoothing needed, because each gradient estimate is already smoother), or equivalently the effective learning rate multiplier is slightly lower than the formula suggests.

In practice, this is rarely a concern because the difference is small, but it is worth knowing when debugging training instability in gradient accumulation setups.

To solidify the mechanics, let us trace through a few steps of momentum by hand on a simple two-parameter problem. Consider a loss function $\mathcal{L}(\theta_1, \theta_2) = 25\theta_1^2 + \theta_2^2$ (a narrow valley with curvature ratio 25:1 between the two dimensions). Starting at $(\theta_1, \theta_2) = (1.0, 0.5)$ with learning rate $\alpha = 0.01$ and momentum $\beta = 0.9$ (PyTorch formulation), initial velocity $\mathbf{v}_0 = (0, 0)$.

The gradient at position $(\theta_1, \theta_2)$ is:

Step 1: Gradient at $(1.0, 0.5)$ is $(50, 1.0)$.

Step 2: Gradient at $(0.5, 0.49)$ is $(25, 0.98)$.

Notice what happened: the velocity in the $\theta_1$ direction (the steep dimension) jumped from 50 to 70, even though the step reduced $\theta_1$ by half. The velocity has overshot and is now pointing in the wrong direction. This will cause oscillations in $\theta_1$ before damping out.

Step 3: Gradient at $(-0.2, 0.471)$ is $(-10, 0.942)$.

The $\theta_1$ velocity is shrinking (from 70 to 53) because the gradient flipped sign. Meanwhile $\theta_2$ velocity is growing from 1.0 to 1.88 to 2.634, because the gradient in $\theta_2$ has been consistently positive. After many steps, the $\theta_1$ velocity will oscillate and decay while $\theta_2$ velocity grows steadily. This is precisely the dampening-oscillations, accelerating-consistent-directions behavior described earlier, made concrete in numbers.

When configuring momentum in PyTorch's torch.optim.SGD, the key parameters are:

• momentum: The momentum coefficient $\beta$. Set to 0.9 as the default starting point. Increase toward 0.99 for smoother loss landscapes or noisier gradients; decrease if you observe unstable training.
• lr: The base learning rate. When using momentum with $\beta = 0.9$, start with a learning rate about 10x smaller than you would use for plain SGD, since the effective rate is amplified.
• nesterov: Set to True to use Nesterov momentum instead of classical Polyak momentum. Requires dampening=0. Often provides slightly faster convergence; rarely hurts.
• dampening: Scaling factor that reduces the gradient contribution to the velocity. Keep at 0 for standard momentum; values above 0 weaken the momentum effect and are rarely used.
• weight_decay: L2 regularization coefficient applied to the parameters, not the velocity. Separate from momentum and can be set independently.

A natural question arises after learning about momentum: if Adam (which combines momentum with adaptive per-parameter learning rates) is available and generally works well, why would you ever use SGD with momentum?

The answer comes down to generalization and compute budget. A growing body of empirical evidence suggests that while Adam converges faster (reaches a low training loss in fewer epochs), SGD with momentum often reaches a lower final test error when given enough training time. This phenomenon is particularly well-documented in computer vision training on ImageNet: the best-performing ResNet and ViT models in the literature are typically trained with SGD plus momentum, not Adam. The intuition is that SGD's less aggressive adaptivity causes it to find wider, flatter minima (often called "sharp" vs. "flat" minima in the generalization literature), and flat minima generalize better to test data.

For natural language processing and transformer-based models, the situation is different. Transformers have highly heterogeneous gradient scales across their layers (embedding layers, attention projections, and feed-forward layers all receive gradients of very different magnitudes), and Adam's per-parameter scaling handles this heterogeneity much better. In practice, virtually all transformer training from BERT to GPT to modern LLMs uses Adam or its variants (AdamW, Adan, etc.) rather than SGD with momentum. The adaptive scaling is necessary in practice rather than optional.

The rule of thumb in modern deep learning is:

• For convolutional neural networks and residual networks on vision tasks with large compute budgets: SGD with momentum is the gold standard.
• For transformer-based language models and multi-modal models: Adam/AdamW with momentum (both classical and second-moment) is the standard.
• For quick prototyping and problem types where you are uncertain: start with Adam, then switch to SGD with momentum if generalization is a concern and you have the compute to train longer.

Understanding momentum thoroughly is essential even when you use Adam, because Adam incorporates a first-moment estimate that is exactly the momentum mechanism described in this chapter, combined with a second-moment estimate for adaptive scaling. The momentum component of Adam is initialized and configured with the same beta1 hyperparameter that corresponds to $\beta$ in classical momentum. Knowing how momentum works makes Adam's hyperparameters interpretable rather than opaque.

Momentum works across many training settings, but it has limitations that practitioners encounter. Understanding these limitations helps you avoid common training failures and decide when to use a different optimizer.

The accumulated velocity that makes momentum effective in flat regions can cause it to overshoot when the loss surface transitions from a flat plateau to a sharp minimum. The optimizer is "moving too fast" as it approaches the minimum and carries past it. This is more pronounced at higher $\beta$ values and can manifest as oscillations around the minimum or even divergence if the minimum is particularly sharp.

Practical mitigation strategies include using a learning rate schedule that reduces the learning rate as training progresses (so the effective step size shrinks near the end), using Nesterov momentum (which applies a corrective force earlier), or reducing $\beta$ late in training. Modern practice often combines all three: start with relatively high $\beta$, schedule the learning rate down, and trust that the reduced learning rate controls the overshoot.

The overshoot problem becomes more severe when the learning rate schedule decreases very rapidly near the end of training. If the learning rate drops by 10x in the final few epochs (a common warm restart strategy), the optimal final position may be much closer to where you started those epochs. Momentum from the early part of the cycle may carry you past the new, closer target. In this regime, resetting the velocity (as described earlier in the warm restart section) combined with the learning rate reset is the safest approach.

Momentum adds one more hyperparameter to tune. The interaction between $\beta$ and the learning rate means that tuning them independently is suboptimal; they should be considered together. The rule of thumb that "effective learning rate = lr / (1 - beta)" helps, but the actual optimization dynamics are more complex, especially on non-quadratic loss surfaces where the curvature changes as the optimizer moves.

A practical approach is to fix $\beta = 0.9$ and only tune the learning rate. This reduces the hyperparameter search to a single dimension while still getting most of the benefit of momentum. If the training loss is noisy, try increasing $\beta$ to 0.95 while halving the learning rate. If you have a learning rate schedule with a large range (initial LR much larger than final LR), be aware that the effective step size amplification is the same at all learning rates, which can make the early part of training disproportionately aggressive.

A fundamental limitation of momentum is that it uses the same velocity smoothing for every parameter. A parameter with a very large gradient and a parameter with a tiny gradient both get the same $\beta$. This means momentum does not adapt to the scale or curvature of individual parameter directions.

In practice, parameters in different layers of a neural network can have wildly different gradient scales. The input embedding layer may receive gradients 100 times smaller than the output layer. Using a single $\beta$ for all parameters means the momentum is calibrated for some parameters but not others. The Adam optimizer, which we cover in the next chapter, addresses this by maintaining per-parameter running statistics that normalize the update scale. This is one of the primary reasons Adam has largely replaced SGD-with-momentum as the default optimizer for many deep learning applications, particularly transformer-based models.

The non-adaptive limitation also means that momentum does not automatically adjust to changes in loss landscape geometry during training. As the optimizer moves from the broad basin of a loss surface into a narrow valley, the appropriate learning rate for each direction changes. Momentum does not detect this shift and must rely on the user's learning rate schedule to compensate. Adaptive methods effectively implement a per-direction, time-varying learning rate schedule automatically, which is a significant advantage in practice.

If the data distribution shifts during training (as in online learning or curriculum learning), the accumulated velocity may push the optimizer in directions that were useful for the old distribution but are harmful for the new one. The momentum "remembers" gradients from data the optimizer is no longer training on, which can cause it to move in the wrong direction for several steps after the shift. Resetting the velocity when the data distribution changes can help in these scenarios.

In curriculum learning, where the training data is presented in a carefully designed order from easy to hard examples, this issue is particularly relevant. The gradients from easy examples early in training may build up velocity in directions that are precisely opposite to what hard examples require later. Managing the velocity reset between curriculum stages is a practical consideration that is easy to overlook.

The effective gradient noise decreases as batch size increases (by the law of large numbers, larger batches give more accurate gradient estimates). When you scale up batch size by a factor of $k$, the gradient noise variance decreases by $k$. This changes the optimal momentum coefficient: with lower noise, you need less smoothing, and you can often use a slightly lower $\beta$ (or equivalently, a slightly higher effective learning rate) when training with large batches.

The "linear scaling rule" for batch sizes (multiply the learning rate by $k$ when multiplying the batch size by $k$) was developed for plain SGD but applies to momentum as well, with the caveat that the warm-up period is especially important at large batch sizes because the initial gradient estimates are very accurate and the natural warm-up from zero velocity is less beneficial.

Despite these limitations, SGD with momentum remains the optimizer of choice for large-scale computer vision training and many language model fine-tuning workflows. Its simplicity, predictability, and well-understood behavior make it a reliable baseline that Adam-based methods must beat to justify their additional complexity. The ImageNet training recipes that achieve state-of-the-art results with ResNets and ViTs predominantly use SGD with momentum, often achieving lower final error than Adam variants when the training budget is sufficient. Momentum is not being retired any time soon.

To complete the intuitive picture, let us visualize how the velocity evolves over the first 50 training steps for a simple one-dimensional optimization problem.

The velocity buildup plot is illuminating. For plain SGD ($\beta = 0$), the velocity is simply the current gradient at each step: it fluctuates with the noisy gradient and never accumulates. For $\beta = 0.9$, the velocity builds up quickly over the first 10 steps and then stabilizes as the optimizer enters the basin around the minimum. For $\beta = 0.99$, the velocity builds up more slowly but reaches a higher peak, requiring the smaller learning rate to prevent the parameter from overshooting too far.

Momentum converts SGD from a reactive optimizer that takes independent steps into a physics-inspired optimizer that accumulates direction and speed across iterations. The key ideas are:

• Momentum maintains a velocity variable $\mathbf{v}_t$ as an exponential moving average of past gradients, controlled by the coefficient $\beta$. In the PyTorch formulation, this is $\mathbf{v}_t = \beta \mathbf{v}_{t-1} + \nabla \mathcal{L}(\theta_{t-1})$.

• In narrow valley landscapes, momentum dampens oscillations across the steep dimensions while accelerating progress along consistent gradient directions toward the minimum. This behavior is equivalent to a low-pass filter on the gradient signal, suppressing high-frequency oscillation noise while preserving low-frequency trend information.

• The effective learning rate for persistent gradient directions is amplified by a factor of $1/(1-\beta)$, which is 10x for the standard $\beta = 0.9$ setting. This amplification requires reducing the base learning rate by the same factor when switching from plain SGD to SGD with momentum.

• Nesterov momentum evaluates the gradient at a lookahead position (the position after applying the committed velocity). This provides an earlier corrective signal and often achieving faster convergence than classical Polyak momentum. It achieves the good $O(1/t^2)$ convergence rate for convex problems. PyTorch implements it at no computational overhead via algebraic reformulation.

• Momentum interacts with learning rate schedules through its natural warm-up behavior (velocity starts at zero and builds up over $1/(1-\beta)$ steps) and through the need to reset velocity at learning rate restarts (warm restarts require momentum reset for best results).

• The primary limitation of momentum is that it is not adaptive: it applies the same smoothing to all parameters regardless of their individual gradient scales. The next chapter introduces Adam, which addresses this by maintaining per-parameter adaptive scaling in addition to the momentum mechanism covered here.

• Despite Adam's advantages for transformer training, SGD with momentum remains the optimizer of choice for large-scale vision model training, often achieving better generalization than Adam when given sufficient training time. Understanding momentum is therefore essential both as a technique in its own right and as a foundation for understanding the first-moment component of Adam and its variants.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about momentum in neural network optimization.