Stochastic Gradient Descent › Saddle Points and Local Minima in High Dimensions

Understanding the geometry of neural network loss landscapes reveals why SGD with noise often works surprisingly well, even without convergence guarantees.

In a neural network with $D$ parameters, the loss surface is a function in $D$-dimensional space. At any stationary point (where the gradient is zero), the local geometry is described by the Hessian matrix $H$, the $D \times D$ matrix of second derivatives:

$$H_{ij} = \frac{\partial^2 \mathcal{L}}{\partial \theta_i \partial \theta_j}$$

The eigenvalues of the Hessian classify the stationary point:

• Local minimum: all eigenvalues are positive. The loss increases in every direction from this point.
• Local maximum: all eigenvalues are negative. The loss decreases in every direction. These are highly unstable and the optimizer neededly never rests here.
• Saddle point: some eigenvalues are positive (uphill directions) and some are negative (downhill directions). The loss can be decreased by moving along any negative eigenvalue direction.

Here is the main insight: in $D$ dimensions, a saddle point requires only that at least one eigenvalue is negative, while a local minimum requires that all $D$ eigenvalues are positive. With $D$ in the billions (for large language models), local minima become exponentially rarer than saddle points. The loss landscape of a deep network is dominated by saddle points.

This property helps SGD. Unlike local minima, saddle points have escape routes: directions in which the loss decreases. If SGD can find and follow these directions, it makes progress even in the absence of a true local minimum.

At a saddle point, the exact gradient is zero, so standard gradient descent stops. Mini-batch SGD has a natural escape mechanism: the random noise in the gradient estimate provides a random perturbation to the update direction. With high probability, this perturbation has a component along the negative-curvature direction (the downhill direction from the saddle), and the optimizer escapes.

The escape time is related to the magnitude of the noise relative to the saddle's curvature. Larger batch sizes mean less noise and slower escape. Smaller batch sizes provide more noise and faster (though less controlled) escape. This is one reason why very large-batch training sometimes gets stuck in poor regions of parameter space that small-batch training avoids.

Among the minima that SGD does find, not all are equally good for generalization. A local minimum can be:

Sharp: the loss surface curves steeply in many directions away from the minimum. A sharp minimum is easy to find but sensitive to small perturbations, which means the model's performance can degrade significantly when the inputs shift slightly from the training distribution.

Flat (wide): the loss surface is nearly flat around the minimum. Many nearby parameter configurations have similar loss. Flat minima tend to generalize better because small distributional shifts in the test data do not dramatically change the loss.

SGD with small batches and appropriate learning rates naturally gravitates toward flat minima. The intuition is that noise in the gradient estimate acts as a random perturbation to the parameters. At a sharp minimum, these perturbations frequently push the optimizer over the steep walls and out of the minimum. At a flat minimum, the perturbations stay within the low-loss region. Over time, SGD spends more time in flat regions, which correspond to models that generalize better.