Stochastic Gradient Descent › Limitations

Despite its central role in deep learning, SGD has real limitations that explain why practitioners have developed more sophisticated optimizers on top of it.

Learning rate sensitivity. Plain SGD is highly sensitive to the learning rate. A rate that is slightly too large causes divergence; a rate that is slightly too small wastes training compute. In contrast to adaptive optimizers, there is no mechanism in plain SGD to automatically compensate for a poor initial choice. You must rely on manual tuning, learning rate range tests, and experience. This sensitivity is particularly acute early in training, when gradient magnitudes can vary wildly across layers. This is why adaptive optimizers like Adam and AdamW, which adjust the effective learning rate per parameter based on gradient history, have largely replaced plain SGD for language model training.

Isotropic updates. SGD applies the same learning rate to all parameters, regardless of how much each one is used or how frequently it receives signal. In natural language processing, this creates a serious problem: token embeddings for common words like "the" or "is" receive gradient updates on nearly every batch, while embeddings for rare tokens like technical jargon may be updated only a handful of times per epoch. When the same learning rate governs both, you face an impossible tradeoff. The rate that works for common tokens overshoots on rare tokens; the rate that works for rare tokens is too slow for common ones. Adaptive optimizers like AdamW address this by scaling updates by the estimated curvature (second moment) of each parameter, effectively giving rare parameters a larger effective learning rate and common parameters a smaller one.

Convergence speed on ill-conditioned problems. If the loss function has very different curvatures in different directions, forming a "ravine" in the loss landscape, SGD oscillates back and forth across the narrow (steep) dimension while making slow progress along the long (shallow) dimension. Mathematically, the condition number of the Hessian measures this ill-conditioning. High condition numbers lead to slow convergence for SGD. Momentum helps with this by accumulating velocity that smooths out oscillations (covered in the next chapter), and adaptive optimizers help even more by effectively pre-conditioning each parameter independently.

Stale gradients in distributed training. Modern language model training runs across hundreds or thousands of GPUs. In distributed training, gradient synchronization takes time: each worker computes its gradient on its local mini-batch, then gradients must be aggregated across all workers before updating the parameters. In asynchronous SGD, the gradient computed by a worker may be based on parameters that have already been updated multiple times by other workers. These "stale" gradients can point in the wrong direction relative to the current parameter values, slowing or destabilizing convergence. Modern distributed training frameworks balance these tradeoffs with techniques like gradient accumulation, local SGD, and carefully tuned communication patterns.

Despite these limitations, SGD remains foundational. Even when using Adam or AdamW, you are using a first-order optimizer that fundamentally estimates the gradient from mini-batches and uses it to update parameters. The core ideas here, batch size, learning rate, schedules, noise, and landscape geometry, all remain directly relevant. Adam and AdamW, which we cover in the next chapters, are best understood as SGD with additional machinery for adapting the learning rate and handling weight decay correctly.