Stochastic Gradient Descent › Summary

Stochastic gradient descent addresses the core impracticality of batch gradient descent on large datasets by replacing the exact gradient with a cheap, unbiased stochastic estimate. The key ideas from this chapter are:

• Mini-batch SGD averages gradients over small random subsets of data, balancing noise against computational efficiency. Batch sizes of 32-512 work well for most NLP tasks.
• Learning rate is the most critical hyperparameter. Too large causes divergence; too small causes slow convergence. Learning rate range tests and standard heuristics (0.01-0.1 for plain SGD, 1e-4 to 3e-4 for transformers) provide starting points.
• Learning rate schedules decay the learning rate over training to transition from exploration to fine-tuning. Cosine annealing with linear warmup is the current default for transformer training.
• SGD noise acts as implicit regularization, preferring flat minima over sharp ones. This helps generalization and allows escape from saddle points.
• High-dimensional loss landscapes are dominated by saddle points rather than local minima. SGD noise provides the random perturbations needed to escape these saddle points.
• Nesterov momentum computes the gradient at a lookahead position rather than the current position, enabling more efficient updates. The next chapter covers momentum in depth.

The chapters that follow build directly on SGD. Chapter 7 (Momentum) shows how to accumulate velocity across steps to accelerate convergence. Chapter 8 (Adam) introduces per-parameter adaptive learning rates that largely solve the sensitivity problem. Chapter 9 (AdamW) fixes Adam's interaction with weight decay, making it the default optimizer for modern language model training.