Large Batch Training › Large Batch Training
Language AI HandbookAll chapters Large Batch Training › Large Batch TrainingOne section · Sections in this chapter1 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Training a neural network means processing data in batches: you take a chunk of examples, compute gradients, and update weights. The batch size you choose is one of the most consequential decisions in the entire training setup. It shapes how quickly you learn, how stable the training is, how well the model generalizes, and how efficiently you use your hardware.
For most of deep learning's history, small to medium batch sizes (32 to 256) were the default. Researchers discovered through painful experience that simply cranking up the batch size made models train faster per epoch but converge to worse solutions. The relationship between batch size and model quality seemed to be a fundamental constraint. Many practitioners accepted this as an immutable law of optimization, choosing their largest tolerable batch size and moving on.
That changed in the mid-2010s, driven partly by practical necessity. As datasets grew to billions of tokens and models reached hundreds of billions of parameters, training on small batches became prohibitively slow. Distributing training across thousands of GPUs required large aggregate batch sizes just to keep the hardware busy. A single forward pass on 4,096 GPUs processes one mini-batch per step; if your batch size is 256, most of your hardware sits idle waiting for gradient synchronization. Researchers had to figure out how to train effectively at scales that were previously unthinkable.
The result was a collection of techniques, with the most important insight being the linear scaling rule: when you multiply the batch size by a factor , multiply the learning rate by the same factor . This rule, backed by empirical and theoretical analysis, made it possible to train models on massive batches without degrading quality. Combined with the LAMB optimizer and careful attention to batch size limits, large batch training became standard practice.
In this chapter, you will understand why batch size affects learning in the way it does, how the linear scaling rule works and why it is justified, where the fundamental limits of batch scaling lie, and how the LAMB optimizer extends these ideas to the pre-training regimes of BERT and GPT.