Stochastic Gradient Descent › Mini-Batch SGD

Mini-batch SGD averages the gradient over a small random subset of the training data, called a batch of size $B$:

$$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \frac{\eta}{B} \sum_{i \in \mathcal{B}_t} \nabla_{\boldsymbol{\theta}} \ell(\boldsymbol{\theta}_t; \mathbf{x}_i, y_i)$$

where:

• $\mathcal{B}_t$: a randomly sampled subset of $B$ training indices at iteration $t$
• $B$: the batch size, typically 32, 64, 128, 256, or 512
• The $\frac{1}{B}$ averaging gives a more stable gradient estimate than a single example

Mini-batch SGD sits between the two extremes. It is noisier than full-batch gradient descent (which uses all $N$ examples) but much cheaper per update. It is less noisy than pure single-example SGD, which tends to produce erratic updates that can destabilize training.

In the deep learning literature, the term "SGD" almost always refers to mini-batch SGD in practice. When you see "SGD with batch size 256", that is mini-batch SGD with $B = 256$.

The choice of batch size involves an engineering tradeoff:

Gradient quality. Larger batches produce more accurate gradient estimates. With $B = N$ (full batch), variance in the gradient estimate is zero. With $B = 1$, variance is maximized. Mini-batches balance variance with cost.

Hardware efficiency. Modern GPUs are designed to process matrices in parallel. A single example cannot saturate a GPU's compute units. Batches of 64-512 examples typically utilize hardware much more efficiently than single-example updates, so larger batches often give more gradient updates per second, not fewer.

Generalization. This is the surprising part. Networks trained with small batches often generalize better to held-out data than networks trained with large batches, even when the training loss is similar. The reasons are still debated, but the leading hypothesis is that small-batch noise acts as a regularizer, pushing the optimizer toward flatter minima that generalize better. We discuss this further in the section on SGD noise below.

Practical observation. Across most NLP tasks, batch sizes in the range of 32 to 512 perform well. Very large batches (above 4096) often require special learning rate warmup strategies (covered below) to maintain convergence quality.