Stochastic Gradient Descent › Key Parameters for SGD
Michael BrenndoerferApril 30, 20251 min read
Language AI HandbookAll chapters Part 10 of 67Neural Network Foundations
Stochastic Gradient Descent › Key Parameters for SGDOne section · Sections in this chapter12 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Understanding the main hyperparameters of SGD helps you tune it effectively for NLP tasks:
- learning_rate: The most critical hyperparameter. Start around 0.01-0.1 for plain SGD, or 1e-3 to 1e-4 for transformers. Use a learning rate finder to narrow the range.
- batch_size: Controls the noise-efficiency tradeoff. Typical values are 32-512. Smaller batches add regularizing noise; larger batches are faster per epoch but may generalize slightly worse.
- momentum: Coefficient for momentum (used when SGD is configured with momentum). Typical value is 0.9. Nesterov momentum is generally preferred over standard momentum.
- weight_decay: L2 regularization applied to weights (separate from learning rate). Typical values are 1e-4 to 1e-2. Helps prevent overfitting, especially in language model fine-tuning.
- schedule: The learning rate schedule. Cosine annealing with linear warmup is the current default for transformer training.
- T_warmup: Number of warmup steps before the main schedule begins. Typically 1-5% of total training steps.