Hyperparameter Selection › Hyperparameter Selection

Training a language model involves far more choices than just the architecture. The learning rate, batch size, weight decay coefficient, warmup schedule, dropout rate, and dozens of other knobs all need values before a single gradient flows. Getting these wrong means wasted compute at best, divergent training at worst. Getting them right can mean the difference between a model that generalizes beautifully and one that memorizes noise.

In the chapters of Part XXXII, we have worked through individual training components in detail: learning rate warmup, decay schedules, large batch training, weight decay, gradient accumulation, and training stability. Now we pull these threads together. Hyperparameter selection is the discipline of choosing sensible values for all of these components jointly, understanding which choices matter most, and building a workflow that lets you iterate efficiently without re-running experiments from scratch every time.

This chapter covers four interconnected ideas. First, how to search for good hyperparameter configurations systematically. Second, how knowledge about good hyperparameters transfers between models of different sizes, enabling you to start large runs with confidence rather than guesswork. Third, which hyperparameters are critical (small deviations cause large performance differences) versus which are robust (performance is nearly flat across a wide range). Fourth, practical default recipes that have emerged from the research community and production practice, giving you a solid starting point for any new training run.