Large Batch Training › Summary

Large batch training changes the statistical properties of gradient estimation. Larger batches provide more accurate gradient estimates (lower variance), enabling the use of larger learning rates that make each step more consequential.

The linear scaling rule states that when you increase the batch size by a factor $k$, you should increase the learning rate by the same factor $k$. This rule emerges from the requirement that the expected weight update after processing $n$ training examples should be the same regardless of batch size. The rule holds well in regimes where the loss landscape is locally smooth but breaks at extremes: very small batches have too much noise, and very large learning rates cause overshooting.

The critical batch size marks the practical upper bound on useful batch scaling. Beyond this size, additional parallelism does not reduce the total number of examples needed for convergence. It represents the point at which gradient noise is no longer the bottleneck for learning.

LAMB extends the linear scaling idea to adaptive optimizers. It combines Adam's per-parameter moment adaptation with a LARS-style layer-wise trust ratio, preventing any layer from taking an unstable step when the global learning rate is large. LAMB enabled BERT pre-training at batch sizes that reduced wall-clock training time from days to minutes on large TPU pods.

Warmup remains essential across all large batch techniques, bridging the early phase where the loss landscape is non-linear and the later phase where the linear scaling assumption holds.

The next chapter on weight decay examines complementary regularization techniques that interact with batch size decisions, completing the toolkit for large-scale training optimization.