Stochastic Gradient Descent › Practical SGD Tuning for NLP
Language AI HandbookAll chapters Stochastic Gradient Descent › Practical SGD Tuning for NLPOne section · Sections in this chapter13 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Training language models has revealed several practical rules that work well across tasks:
Pre-training transformers. Use AdamW (covered in its own chapter), not plain SGD. AdamW combines adaptive learning rates (per-parameter) with proper weight decay. For learning rates, use cosine decay from 1e-4 to 3e-4 with linear warmup for 1-2% of steps.
Fine-tuning transformers. Smaller learning rates than pre-training, typically 1e-5 to 1e-4. Cosine decay with warmup still applies. Larger batches (256-2048) are common when fine-tuning on labeled datasets.
Classical NLP models. Plain SGD with momentum 0.9 works well for simpler models like logistic regression and shallow networks. Learning rates of 0.01-0.1, batch sizes of 32-128. Step decay (halve every 10 epochs) or cosine annealing both work.
Monitoring training. Always track both training and validation loss. If validation loss stops improving while training loss continues to decrease, you are overfitting and may want to reduce the learning rate more aggressively, increase weight decay, or stop early.
Gradient norms. Monitor the gradient norm (the Euclidean norm of all gradients stacked into a vector) throughout training. A gradient norm that grows without bound signals divergence. A gradient norm that drops to near zero too quickly signals premature convergence. The next chapter on Gradient Clipping covers this in detail.