Hyperparameter Selection › Hyperparameter Search

Hyperparameter search is the process of finding configurations that perform well by evaluating multiple candidates. The right search strategy depends on your computational budget, the number of hyperparameters you are tuning, and how smooth the loss landscape is as a function of those parameters.

Grid search evaluates every point on a regular grid over the hyperparameter space. If you want to test three learning rates and four weight decay values, grid search runs twelve experiments covering all combinations. It is exhaustive and reproducible, with results that are easy to reason about.

The problem is combinatorial explosion. Adding one more dimension to search multiplies the number of experiments by the number of values in that dimension. A grid over six hyperparameters with five values each requires $5^6 = 15{,}625$ experiments. This is practical only for one or two hyperparameters or when each experiment is very cheap.

Grid search also wastes resources on less important dimensions. If learning rate matters a lot and weight decay barely matters, half your grid experiments vary weight decay while holding learning rate fixed at a bad value. You get many evaluations at useless configurations. The structure that makes grid search appealing, its exhaustiveness over the chosen axes, becomes a liability when the axes are not equally important, which is virtually always the case in practice.

Random search samples configurations uniformly at random from the hyperparameter space. Instead of a regular grid, you get scattered points. This sounds less systematic, but it is typically more efficient than grid search for the following reason: most hyperparameter landscapes have low intrinsic dimensionality. A few dimensions matter a lot and the rest matter little. In a grid, you spend $1/N$ of your evaluations on each unique value of an important dimension; in random search, every evaluation uses a different value of that important dimension, so you explore it more thoroughly.

The seminal result from Bergstra and Bengio (2012) showed that random search finds equally good or better configurations than grid search using far fewer evaluations, especially when the number of relevant hyperparameters is small relative to the total number being tuned. For practical LLM training, where you might have fifteen hyperparameters but only three or four that substantially affect the outcome, random search is a natural choice.

Random search is embarrassingly parallel: all trials are independent and can run simultaneously. With a fixed budget of, say, 50 experiments, you can evaluate all 50 in parallel if you have the hardware. This property is invaluable in large-scale LLM work, where parallelizing across many GPUs is the norm.

The main limitation of random search is that it does not learn from previous evaluations. Each new sample is drawn independently of all past results, so random search cannot refine its search around promising regions. If the loss landscape has a sharp, narrow optimum, you may need many samples to hit it by chance. This observation motivates the next step in the progression: adaptive search methods that use past results to guide future queries.

Bayesian optimization treats hyperparameter search as a sequential decision problem. After each experiment, it updates a probabilistic model of how performance depends on the hyperparameter configuration and uses that model to choose the next configuration to evaluate. The key idea is to balance exploitation (trying configurations near the best found so far) with exploration (trying configurations in regions we know little about).

The probabilistic model is typically a Gaussian process or a tree-structured Parzen estimator (TPE). The model outputs a predicted performance and an uncertainty estimate. The next configuration is chosen by maximizing an acquisition function that combines expected performance with uncertainty. Common acquisition functions include:

• Expected Improvement (EI): the expected amount by which the next trial improves over the current best
• Upper Confidence Bound (UCB): predicted performance plus a scaled uncertainty term
• Thompson Sampling: sample one function from the posterior and maximize it

Each of these acquisition functions encodes a different strategy for balancing exploitation and exploration. EI focuses on configurations likely to beat the current best. UCB adds a bonus for poorly-explored regions. Thompson Sampling introduces randomness by sampling a plausible objective function and optimizing it, which naturally diversifies exploration without needing a manually tuned exploration coefficient.

Bayesian optimization tends to find good configurations in fewer evaluations than random search, particularly for expensive objectives. The cost is that each selection step requires fitting a surrogate model and solving an inner optimization. For problems where each evaluation costs hours, this overhead is negligible.

The limitation is sequential dependency: Bayesian optimization selects one configuration at a time based on previous results. In practice, most teams run experiments in parallel batches, which breaks the strict sequential assumption. Asynchronous variants like hyperopt and Optuna handle this by selecting multiple candidates using the current surrogate model without waiting for all running trials to finish. They use a "lie" strategy: they pretend that currently running trials have already returned some expected result and use this fictitious observation to select the next batch of candidates.

For LLM training, the high cost of individual evaluations makes Bayesian optimization appealing in principle, but the small number of feasible trials (often fewer than 30 at a given scale) limits the surrogate model's accuracy. In this regime, the advantage over random search is modest, and many practitioners use random search for simplicity with Bayesian optimization reserved for settings where even a few evaluations is prohibitively expensive.

Even with efficient search strategies, each trial trains a full model to completion before reporting its final performance. This is wasteful: a configuration with a learning rate ten times too large will diverge within the first thousand steps, yet a naive approach spends a full training run confirming that fact.

Early stopping cuts this waste by terminating runs that fall behind early. At fixed intervals during training, you compare all running configurations; those that perform significantly below the current best are terminated, and their compute is reallocated to the more promising survivors. This is not always safe (some good configurations start slowly before finding their footing) but works well when early performance correlates with final performance, which is generally true for learning rate sensitivity.

Successive halving formalizes early stopping into a systematic elimination protocol. It dramatically reduces cost by training all candidates for a short period, discarding the worst half, doubling the resources for the survivors, and repeating until one winner remains.

Formally, given $n$ initial configurations and a total budget $B$, successive halving proceeds as:

$$\begin{aligned} &\text{Round 1: train all } n \text{ configs for } B/(n \log_2 n) \text{ steps, keep top } n/2 \\ &\text{Round 2: train surviving } n/2 \text{ configs for } 2B/(n \log_2 n) \text{ steps, keep top } n/4 \\ &\vdots \\ &\text{Round } \log_2 n: \text{ train final 2 configs, return winner} \end{aligned}$$

where:

• $n$: the number of initial configurations
• $B$: total compute budget in steps (or flops)
• $\log_2 n$: number of elimination rounds

The assumption is that good configurations are identifiable early in training. This holds reasonably well for learning rate (divergent runs fail quickly) but less well for regularization (the effect of weight decay often only appears after substantial training). When performance rankings correlate between early and late training, successive halving finds nearly-optimal configurations with a fraction of the total compute.

Hyperband extends successive halving by running it with multiple values of $n$, trading off between getting more configurations (wide, shallow search) and training each longer (narrow, deep search). With a large $n$, you screen many candidates quickly but eliminate promising slow-starters too early. With a small $n$, each survivor gets more training but you start with fewer candidates. Hyperband hedges by running multiple brackets simultaneously, each with a different $n$, then selecting the best result across all brackets. This makes Hyperband more robust to the assumption that early performance predicts final performance.

The practical significance of Hyperband is substantial. In a controlled comparison, Hyperband with a total compute budget of $B$ frequently outperforms running random search for the same $B$ on a single configuration, because it can explore far more of the hyperparameter space in the early rounds while still committing full resources to the best candidates. For LLM pretraining where individual runs are expensive, Hyperband-style thinking should guide how you allocate your search budget.

For large language model training, a practical search protocol takes into account that individual runs are expensive and that a small number of hyperparameters dominate performance:

1. Identify proxy tasks: Use a small model (a few hundred million parameters at most) trained on a representative data sample for a small number of steps. If the proxy correlates with full-scale performance, this drastically reduces search cost.

2. Fix cheap defaults first: Set architecture-linked hyperparameters (layer count, hidden size, attention heads) by scaling laws. Fix dropout to 0 for the pretraining phase. These are not search dimensions.

3. Search learning rate: This is the most critical hyperparameter. Run random search over two to three orders of magnitude around a reasonable prior (discussed below in defaults). Use the proxy model and early stopping.

4. Search weight decay: After finding a good learning rate, search weight decay in a one-dimensional sweep.

5. Search batch size: Batch size interacts with learning rate. If you change batch size substantially, re-verify the learning rate.

6. Verify on target scale: Confirm the best configuration found at proxy scale transfers to the target scale before committing to a full training run.