Hyperparameter Selection › Hyperparameter Interactions

Before diving into transfer and defaults, it is worth pausing on a nuance that the simple sensitivity analyses above tend to obscure: hyperparameters interact with each other in non-trivial ways. The optimal learning rate is not independent of batch size. The appropriate warmup duration depends on the schedule type. The effective regularization from weight decay depends on the total number of training steps. Understanding these interactions prevents you from fixing one hyperparameter at a suboptimal value while searching for another, which would cause you to consistently underestimate the true achievable performance.

When you scale the batch size, the gradient at each step becomes an average over more samples. The variance of this gradient estimate decreases proportionally. A larger batch gives a more accurate estimate of the true gradient direction, which allows taking a larger step without overshooting. This motivates the linear scaling rule: if you double the batch size, doubling the learning rate maintains approximately the same effective noise-to-signal ratio in the gradient updates.

The linear scaling rule works reliably for moderate batch sizes (up to roughly 32,000 tokens per batch in LLM training) but breaks down at very large batches. When the batch is so large that the gradient estimate is nearly noise-free, the model is in the data-limited regime and increasing the batch further does not help, so scaling the learning rate linearly would make steps too large. The square root scaling rule, which multiplies the learning rate by $\sqrt{k}$ when scaling the batch by $k$, is more conservative and tends to be safer in this regime. Most practitioners use the linear rule up to the critical batch size identified by Kaplan et al. (2020), then switch to the square root rule for larger batches.

The role of warmup is to stabilize the Adam optimizer's second moment estimate before taking large steps. At the beginning of training, the second moment estimate is initialized to zero and has seen very few gradient samples. If the learning rate is already at its peak value during this initialization phase, the effective step sizes for parameters with small gradient history can be wildly large, causing early instability. Warmup suppresses this by holding the learning rate low while the second moment estimate builds up.

The appropriate warmup duration depends on $1/(1 - \beta_2)$, the effective window of the second moment estimate. With $\beta_2 = 0.999$, the window spans approximately 1000 steps, so roughly 1000 steps of warmup are needed for the estimate to stabilize. With $\beta_2 = 0.95$, the window spans only 20 steps, so much shorter warmup suffices. Using $\beta_2 = 0.95$ (the LLaMA default) with a 2000-step warmup provides comfortable margin; using $\beta_2 = 0.999$ with only 500 steps of warmup risks early instability.

This interaction is often overlooked when copying hyperparameters from one recipe to another. If you take the LLaMA recipe ($\beta_2 = 0.95$, 2000 steps warmup) and change only $\beta_2$ to $0.999$ (perhaps because your framework uses it as default), you should also increase the warmup duration to match.

Weight decay applies a shrinkage force proportional to each parameter's current value at every step. Over a training run of $T$ steps with learning rate $\eta$ and weight decay coefficient $\lambda$, a parameter that receives no gradient update will be multiplied by $(1 - \eta \lambda)^T$ total. With $\eta = 3 \times 10^{-4}$, $\lambda = 0.1$, and $T = 100{,}000$ steps:

$$(1 - \eta \lambda)^T = (1 - 3 \times 10^{-5})^{100000} \approx e^{-3} \approx 0.050$$

A weight that receives no gradient signal will shrink to about 5% of its initial value over this training run. This is a powerful shrinkage. For longer runs or higher weight decay, the shrinkage is even more dramatic. If you extend a training run from 100K to 500K steps without adjusting weight decay, the effective regularization becomes five times stronger. This can suppress the model's capacity to represent certain patterns. The implication is that when you change training duration significantly, you should re-examine the weight decay setting. A run five times longer than a reference run may benefit from weight decay reduced by roughly a factor of five.