Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Hyperparameter Selection › Critical vs. Robust Hyperparameters

Michael BrenndoerferJanuary 29, 20268 min read

Language AI HandbookAll chapters

Part 32 of 67Training Optimization

Chapter 238Hyperparameter Selection › Critical vs. Robust Hyperparameters

Reading section 6 of 14 · Critical vs. Robust Hyperparameters · 8 minSection 6 of 14 · 8 min

Section 6 of 14 · 8 min read

Hyperparameter Selection › Critical vs. Robust HyperparametersOne section · Sections in this chapter6 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Not all hyperparameters are equal in their impact on model quality. Understanding which hyperparameters are critical (meaning small deviations from the optimal value cause large performance degradation) versus robust (meaning performance is nearly flat across a wide range) tells you where to invest search effort.

A Taxonomy of SensitivityLink Copied

Critical hyperparameters are those where the performance landscape has a sharp peak. The loss drops steeply on either side of the optimum. For these, you need a precise value or at least a value within a narrow band.

Reliable hyperparameters are those where the performance field is nearly flat. The loss changes little across orders of magnitude. For these, a reasonable default suffices and extensive search is wasteful.

Between these extremes are hyperparameters that matter moderately: performance degrades gradually as you move away from the optimum, but you can tolerate some imprecision.

The taxonomy is:

Critical (search carefully):

Learning rate: The most important hyperparameter in practice. Performance is highly sensitive to learning rate, with an optimum that can be 2-3 orders of magnitude wide in log-space but still has a factor-of-2 sensitivity near the edges. A learning rate that is 5x too large often causes divergence; one that is 5x too small trains stably but reaches poor final performance.
Learning rate schedule: The shape of the decay curve significantly affects final performance. Cosine decay outperforms step decay and linear decay in most settings; using no decay at all loses 5-15% relative perplexity.

Moderate sensitivity (use informed defaults, search if budget allows):

Weight decay: Performance degrades smoothly outside the optimal range. Values between $0.01$ and $0.1$ are generally safe; values below $0.001$ often lead to overfitting and values above $0.3$ can hurt convergence.
Warmup duration: A small amount of warmup (1-5% of total steps) matters; the precise value within this range does not.
Batch size: Affects sample efficiency but not final performance dramatically if learning rate is adjusted. Too small (below effective batch size) hurts; too large wastes compute without benefit.

Robust (use universal defaults):

Adam $\beta_1$ : Values between $0.85$ and $0.95$ all work nearly identically. Default $0.9$ is fine everywhere.
Adam $\beta_2$ : Values between $0.95$ and $0.999$ all work nearly identically. Default $0.999$ is the most common; $0.95$ is used for training stability.
Adam $\varepsilon$ : Has essentially no effect on performance across a wide range ( $10^{-8}$ to $10^{-6}$ ) except at extreme values.
Gradient clipping threshold (when using a reasonable value): Values between $0.5$ and $5.0$ give nearly identical results.

In[10]:

Code

import numpy as np

rng = np.random.default_rng(0)


def model_perplexity(lr, wd, beta2, warmup_frac, noise=0.02):
    """
    Simulate how final model perplexity depends on different hyperparameters.
    Returns a perplexity value (lower is better).

    Critical: lr (sharp optimum at log10=-3.5)
    Moderate: wd (optimum at log10=-2, gradual)
    Robust: beta2 (nearly flat from 0.95 to 0.999)
    Moderate: warmup_frac (flat from 0.01 to 0.05)
    """
    # Learning rate component: sharp quadratic in log space
    lr_opt = np.log10(3e-4)
    lr_comp = 15.0 * (np.log10(lr) - lr_opt) ** 2

    # Weight decay component: moderate quadratic in log space
    wd_opt = np.log10(1e-2)
    wd_comp = 2.0 * (np.log10(wd) - wd_opt) ** 2

    # Beta2 component: very flat
    beta2_comp = 0.3 * (beta2 - 0.999) ** 2 * 100

    # Warmup component: moderate but only for very low values
    warmup_comp = 1.5 * max(0, 0.005 - warmup_frac) ** 2 * 10000

    base_perplexity = 20.0
    noise_term = rng.normal(0, noise)
    return (
        base_perplexity
        + lr_comp
        + wd_comp
        + beta2_comp
        + warmup_comp
        + noise_term
    )


# Vary each hyperparameter while keeping others at their optimum
lr_values = np.logspace(-5, -2, 50)
wd_values = np.logspace(-4, -1, 50)
beta2_values = np.linspace(0.90, 0.999, 50)
warmup_values = np.linspace(0.001, 0.10, 50)

lr_perplexities = [
    model_perplexity(lr=v, wd=1e-2, beta2=0.999, warmup_frac=0.02)
    for v in lr_values
]
wd_perplexities = [
    model_perplexity(lr=3e-4, wd=v, beta2=0.999, warmup_frac=0.02)
    for v in wd_values
]
beta2_perplexities = [
    model_perplexity(lr=3e-4, wd=1e-2, beta2=v, warmup_frac=0.02)
    for v in beta2_values
]
warmup_perplexities = [
    model_perplexity(lr=3e-4, wd=1e-2, beta2=0.999, warmup_frac=v)
    for v in warmup_values
]

# Compute sensitivity: range of perplexity change
lr_range = max(lr_perplexities) - min(lr_perplexities)
wd_range = max(wd_perplexities) - min(wd_perplexities)
beta2_range = max(beta2_perplexities) - min(beta2_perplexities)
warmup_range = max(warmup_perplexities) - min(warmup_perplexities)

Out[11]:

Console

Hyperparameter sensitivity (perplexity range across full sweep):

  Learning rate:   34.79 perplexity points  -> CRITICAL
  Weight decay:    8.03 perplexity points  -> MODERATE
  Beta2:           0.34 perplexity points  -> ROBUST
  Warmup fraction: 0.28 perplexity points  -> MODERATE

Search priority: focus compute on critical hyperparameters.

The sensitivity analysis confirms the intuition: learning rate has the largest impact on performance, weight decay and warmup fraction have moderate impacts, and beta2 has minimal impact. This tells you how to allocate your search budget.

Out[12]:

Visualization

Two perplexity curves over log10 hyperparameter values. The learning-rate curve forms a steep U shape around minus 3.5, while the weight-decay curve forms a broader, shallower U around minus 2. — Learning rate produces a sharp perplexity minimum near 3e-4, while weight decay has a much shallower basin near 1e-2. Moving the learning rate tenfold from its optimum costs roughly 15 perplexity points, making it the higher-priority search dimension.

Out[13]:

Visualization

Dual-axis line plot comparing beta2 and warmup sweeps. The beta2 curve changes gradually by less than half a perplexity point, and the warmup curve quickly flattens after 0.5 percent. — Adam beta2 changes simulated perplexity by less than half a point across the displayed range, while warmup incurs a penalty only below 0.5 percent of training. The broad flat regions explain why defaults are usually sufficient for both parameters.

Sensitivity and Model ScaleLink Copied

Sensitivity patterns are not always constant across model scale. Some observations from large-scale training:

Learning rate sensitivity increases with model scale. Larger models are more sensitive to suboptimal learning rates, partly because their optimization landscapes have more complex structure.
Weight decay sensitivity also increases somewhat with scale. Larger models have more capacity to overfit, making regularization more important.
Beta2 sensitivity is scale-invariant in practice. The same defaults work from small models to the largest trained.
Dropout sensitivity follows a U-curve: too little and the model overfits; too much and capacity is wasted. The optimal dropout rate tends to increase with model size but is relatively robust within a factor of 2.

The implication for your search workflow is to concentrate your compute on the learning rate first, then weight decay, and treat everything else as fixed defaults. Within a 5x learning rate range around your prior, you should be able to find a near-optimal value in ten to twenty proxy experiments. Within a 3x weight decay range around $0.1$ , five experiments usually suffice. You will recover far more performance from careful learning rate tuning than from fine-grained optimization of beta2 or epsilon.