Hyperparameter Selection › Summary

Hyperparameter selection in LLM training requires choosing configurations for learning rate, schedule, optimizer parameters, regularization, and batch size simultaneously. The key ideas from this chapter are:

• Grid search covers all combinations but scales poorly with dimensionality. Random search explores the important dimensions more efficiently at the same compute budget.
• Bayesian optimization finds good configurations in fewer evaluations than random search by building a surrogate model of the objective.
• Successive halving and Hyperband reduce total compute by eliminating poor configurations early in training, allocating more resources to promising candidates.
• Hyperparameter interactions matter: learning rate and batch size scale together, warmup duration depends on $\beta_2$, and weight decay interacts with training duration. Fix these jointly rather than independently.
• Hyperparameter transfer exploits the observation that good configurations on small models predict good configurations on large models. This works reliably under muP for learning rate across model widths.
• Critical hyperparameters (learning rate, schedule shape) require careful search. Robust hyperparameters (Adam betas, gradient clipping) work with universal defaults across nearly all settings.
• Default recipes (GPT-style, Chinchilla, LLaMA-style) provide validated starting points. Use the size-based learning rate prior from Table lr-by-size and adjust based on your specific setup.
• Fine-tuning requires lower learning rates, shorter warmup, and lower weight decay than pretraining, because the model is adapting an existing representation rather than building one from scratch.
• The recommended workflow is: set architecture hyperparameters by scaling laws, search learning rate on a proxy model, transfer to the target, and monitor gradient norms and loss curves during training.