Stochastic Gradient Descent › Stochastic Gradient Descent

In the previous chapter on backpropagation, we learned how to compute the gradient of the loss with respect to every parameter in a neural network. We have the gradient. Now what do we do with it?

The straightforward answer is gradient descent: repeatedly move parameters in the direction opposite to the gradient. But this naive formulation hides a critical practical problem. Computing the exact gradient requires evaluating the loss over your entire training dataset. If you have 10 million training examples, you need to run all 10 million through the network before taking a single parameter update step. This is batch gradient descent, and on modern datasets it is painfully slow.

Stochastic gradient descent (SGD) is the solution. Instead of computing the exact gradient over the full dataset, SGD estimates the gradient using a single training example or a small random batch. The estimate is noisy, but it comes fast, and with careful tuning, that noise turns out to be a feature rather than a bug.

SGD is the engine that makes deep learning work in practice. Every major language model you interact with today, from BERT to GPT-4, was trained using SGD or one of its descendants (Adam, AdamW). Understanding SGD at a deep level means understanding why neural networks can be trained at all, how learning rate choices shape the quality of what you train, and why certain tricks like mini-batching and learning rate schedules are so universally adopted. We will explore all of this here. The next two chapters will build on SGD to cover momentum and the Adam optimizer, which add adaptive mechanisms on top of the core ideas introduced here.