Stochastic Gradient Descent › Learning Rate Selection

The learning rate $\eta$ is the most important hyperparameter in SGD. Choose it badly and your model will diverge, oscillate, or converge too slowly to be useful. Understanding the mechanics of learning rate selection is one of the most transferable skills in deep learning.

Consider what different learning rates do to the optimization trajectory.

Too large. If $\eta$ is too large, each step overshoots the local minimum. The loss oscillates or increases instead of decreasing. In extreme cases, the gradient explodes: parameter values reach infinity or NaN within a few steps.

Too small. If $\eta$ is too small, the optimizer makes tiny steps and takes a very long time to converge. Training becomes inefficient. With a decaying learning rate schedule, an initial rate that is too small can prevent you from ever reaching a good region of parameter space.

Just right. A good learning rate produces steady, consistent loss reduction without oscillation. The loss curve slopes downward smoothly.

Practitioners have developed several heuristics for choosing a learning rate:

• Start around 0.01 to 0.1 for standard SGD. These values work well across many tasks.
• Use the learning rate range test. Train for a few hundred steps while increasing the learning rate exponentially from $10^{-7}$ to 1, and plot the loss versus learning rate. The optimal learning rate is usually just before the point where the loss starts to increase.
• If using larger batches, scale the learning rate. A commonly used rule is linear scaling: if you double the batch size, double the learning rate. The intuition is that with twice as many examples per step, each step produces a more accurate gradient, so you can afford to take larger steps.
• Try 3e-4 for Adam-based optimizers. Adam is generally less sensitive to learning rate than plain SGD, and 3e-4 (also written 0.0003) works surprisingly often across language model pre-training tasks.

For plain SGD on NLP tasks, learning rates between 0.01 and 0.1 are common. For pre-training transformers, much smaller rates are typical (1e-4 to 1e-3) because the gradient landscape is sharper.