Hyperparameter Selection › Limitations and Impact

Hyperparameter search remains expensive despite methodological advances. Even with muP enabling transfer, you must still run experiments on the proxy model, which takes time and compute. For very large models (100B parameters and above), even the proxy is expensive: a "small" proxy at 1/100th the parameter count might still be a 1B parameter model. In practice, most large-scale training runs at frontier scale use informed defaults with light verification, not exhaustive search. The real value of systematic search methods is at the medium scale (100M to 10B parameters) where you have enough budget to explore but not so much that defaults always suffice.

The assumption underlying hyperparameter transfer is that the proxy model behaves qualitatively like the target model. This holds for width scaling under muP but is less reliable for depth scaling, for qualitative differences in data distribution, or for different training objectives. Pretraining and fine-tuning have different optimal hyperparameters: fine-tuning typically benefits from lower learning rates (often 1/10th to 1/100th of the pretraining rate), shorter schedules, and lower weight decay. When these assumptions break down, configurations that seemed optimal on the proxy can underperform on the target by a substantial margin, and you will not know until you have committed significant compute.

A deeper limitation is that muP is not yet universally adopted. Implementing muP correctly requires modifying the initialization and learning rate scaling for each parameter type, which is non-trivial in existing codebases and easy to get wrong. Some publicly available implementations have subtle bugs. Until muP is a standard feature of training frameworks, many practitioners work with standard parametrization and rely on the empirical learning rate priors described in Table lr-by-size combined with a small amount of verification at the target scale.

Default recipes embed assumptions about the training regime. The GPT-style recipe assumes long-horizon training on diverse text data. Applying it to a domain-specific model trained for far fewer steps or on narrow data may give suboptimal results. Weight decay of $0.1$ is appropriate when the dataset is large relative to model capacity. It can over-regularize when fine-tuning a large model on a small dataset, suppressing the model's ability to adapt to the target domain. In such cases, reducing weight decay to $0.01$ or even $0.0$ for the fine-tuning phase is often beneficial.

Despite these limitations, the development of systematic search strategies, muP for transfer, and community-validated default recipes has substantially reduced the cost of training new models. Researchers today can configure a training run with much more confidence than those five years ago, not because the search problem is solved, but because the prior over good configurations has narrowed dramatically. The field has moved from "search everything" to "know what to search, use defaults for everything else." The next chapters move to Part XLVII, where we explore specialized training for code generation models, which inherits this foundation but adds new considerations around tokenization and training objectives specific to programming languages.