Stochastic Gradient Descent › The Problem with Batch Gradient Descent

To understand why SGD matters, we first need to see what goes wrong with full-batch gradient descent on large datasets.

In standard gradient descent, we compute the loss averaged over all $N$ training examples and then update the parameters:

$$\mathcal{L}(\boldsymbol{\theta}) = \frac{1}{N} \sum_{i=1}^{N} \ell(\boldsymbol{\theta}; \mathbf{x}_i, y_i)$$ $$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta \nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta}_t)$$

where:

• $\boldsymbol{\theta}$: the full parameter vector of the network
• $\ell(\boldsymbol{\theta}; \mathbf{x}_i, y_i)$: the loss on a single training example $(\mathbf{x}_i, y_i)$
• $N$: the total number of training examples
• $\eta$: the learning rate, a positive scalar controlling step size
• $t$: the iteration index

The gradient $\nabla_{\boldsymbol{\theta}} \mathcal{L}$ is an exact average over the full training set. This is mathematically clean: the gradient points precisely downhill in the loss landscape. But it has three serious problems in practice.

Computational cost. Modern NLP datasets contain millions of examples. Each forward-backward pass is expensive. Waiting until you have seen all $N$ examples before updating means you update parameters just once per epoch. With a dataset of 100 million sentences, you might wait hours between updates.

Memory cost. Full-batch gradient descent requires holding the gradients from all examples in memory before computing the average. For large datasets, this can exceed the available GPU memory by orders of magnitude.

Escape from saddle points. Neural network loss landscapes are high-dimensional and filled with saddle points (more on this later in this chapter). The exact gradient at a saddle point is exactly zero, which means batch gradient descent stalls. A little noise can kick the optimizer free.

The key insight of SGD is that we don't need the exact gradient. We need a gradient estimate that points, on average, in the right direction. A single randomly chosen training example provides exactly that:

$$\nabla_{\boldsymbol{\theta}} \ell(\boldsymbol{\theta}; \mathbf{x}_i, y_i) \approx \nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta})$$

The left side is the gradient on example $i$, a cheap computation. It is a noisy estimate of the true gradient on the right, but its expectation equals the true gradient:

$$\mathbb{E}_{i \sim \text{Uniform}(1, \ldots, N)}\left[\nabla_{\boldsymbol{\theta}} \ell(\boldsymbol{\theta}; \mathbf{x}_i, y_i)\right] = \nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta})$$

where the expectation is taken over a uniformly random choice of example index $i$. This is the mathematical foundation of SGD: stochastic gradients are unbiased estimators of the true gradient. Each individual step is noisy, but over many steps the noise averages out.