Hyperparameter Selection › Why Hyperparameter Selection is Hard

The challenge is not simply that there are many hyperparameters. The challenge is that they interact. A learning rate that works well with a batch size of 256 may cause divergence at batch size 4096. Weight decay that prevents overfitting at one scale may cripple convergence at another. Warmup duration that is appropriate for a cosine schedule may be inadequate for a constant rate. The search space is high-dimensional, the objective function is expensive to evaluate (each evaluation requires training a model), and the interactions between dimensions create a landscape with local optima and flat plateaus that are hard to search.

Compounding this, the feedback loop is slow. Unlike tuning a software system where you can run hundreds of tests per hour, training a large language model takes days or weeks. Evaluating 100 hyperparameter configurations on a model with 7 billion parameters is not feasible. This computation cost has driven researchers toward three complementary strategies: systematic search on small proxy models, theoretical understanding of how hyperparameters scale, and empirical default recipes derived from the collective experience of many training runs.

There is also a subtler challenge: the objective landscape you observe during short proxy runs may not match the landscape you care about. A configuration that minimizes loss after 5,000 steps may not be the one that minimizes loss after 100,000 steps. Regularization effects take time to manifest. The optimal weight decay for a long run is often higher than the optimal weight decay for a short run, because the model has more time to overfit. These temporal dependencies make proxy experiments informative but not perfectly predictive of full-scale results.