Hyperparameter Selection › Default Recipes

The research community has converged on several default recipes for LLM training. These represent the distilled experience of many training runs and are reliable starting points for new models. You will still need to tune learning rate and verify on your specific setup, but using these defaults dramatically narrows the search space.

The configuration used in GPT-3 and refined in subsequent work has become the de facto standard for autoregressive LLM pretraining:

• Optimizer: Adam with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\varepsilon = 10^{-8}$
• Learning rate: peak LR in the range $[10^{-4}, 3 \times 10^{-4}]$ depending on model size, scaled by $1/\sqrt{d_{model}}$ approximately
• Learning rate schedule: cosine decay to 10% of peak LR
• Warmup: linear warmup for 1% to 4% of total training steps (up to a few thousand steps)
• Weight decay: $0.1$ applied to all non-bias, non-embedding parameters
• Gradient clipping: 1.0 (global gradient norm)
• Batch size: large, typically 0.5M to 4M tokens per batch
• Dropout: 0 during pretraining (all regularization via weight decay and data diversity)

Note the choice of $\beta_2 = 0.95$ rather than the Adam default of $0.999$. This reflects the observation that large-batch LLM training benefits from a shorter memory in the second moment estimate. With $\beta_2 = 0.999$, the second moment estimate is dominated by a long history of gradients, which can cause the optimizer to respond too slowly to sudden changes in gradient magnitude during training. The value $0.95$ gives a more responsive effective window of approximately $1/(1 - 0.95) = 20$ steps rather than 1000 steps.

The decision to set dropout to zero during pretraining is deliberate. Dropout was a critical regularizer in the pre-LLM era, when models were trained on relatively small, task-specific datasets. For LLM pretraining on trillions of tokens from diverse sources, data diversity itself provides strong implicit regularization. Adding dropout during pretraining hurts sample efficiency without providing meaningful overfitting protection. Dropout is sometimes reintroduced during fine-tuning, where the dataset is smaller and overfitting becomes a real concern.

The Chinchilla paper (Hoffmann et al., 2022) contributed scaling laws and practical training defaults for compute-optimal models:

• Dataset size: train for roughly $20 \times N$ tokens where $N$ is the model size in parameters
• Learning rate: similar to GPT-style but with slightly lower weight decay ($0.01$ to $0.1$ depending on scale)
• Cooldown: explicit cooldown phase of 10% of total steps before linear decay ends

The important takeaway from Chinchilla for hyperparameter selection is that dataset size and model size are themselves hyperparameters that interact. Under-training a large model (fewer tokens than compute-optimal) or over-training a small model both give worse performance per flop.

The Chinchilla result says both the number of parameters $N$ and the number of training tokens $D$ should scale equally with compute. Empirically, $D \approx 20N$ (train on 20 tokens per parameter). The compute cost of training a transformer is approximately $C \approx 6ND$ FLOPs (roughly 6 multiply-accumulates per parameter per token). Substituting $D = 20N$:

$$\begin{aligned} C &\approx 6 N D = 6 N \cdot 20N = 120 N^2 \\ N^* &= \sqrt{\frac{C}{120}} \end{aligned}$$

where:

• $N^*$: optimal model size in parameters for a given compute budget
• $D^*$: optimal number of training tokens, approximately $20N^*$
• $C$: total compute budget in floating-point operations (FLOPs)

(The exact constant varies by source; the Chinchilla paper uses coefficients derived from empirical fits. The key insight is that $N^*$ scales as $\sqrt{C}$, meaning you should grow both model size and data together as compute increases.)

For a given compute budget, training a model with $N^*$ parameters is more efficient than training a larger model for fewer steps or a smaller model for more steps.

The Chinchilla compute-optimal recipe optimizes training loss at a fixed compute budget. In deployment, the cost of inference often dominates the cost of training: a model that you run millions of times at inference may justify training for longer (more tokens than Chinchilla-optimal) to push down the model size needed to achieve a target quality. This trade-off has led several organizations to train smaller models for longer than the Chinchilla recipe recommends, accepting higher training cost in exchange for lower inference cost.

The LLaMA family (Touvron et al., 2023) refined the GPT recipe with several changes that have become widely adopted:

• Optimizer: AdamW (weight decay decoupled from the gradient update, as discussed in the Weight Decay chapter)
• Learning rate: $3 \times 10^{-4}$ for 7B, $1.5 \times 10^{-4}$ for 65B
• Schedule: cosine decay to $10^{-5}$ (nearly to zero)
• Warmup: 2000 steps (roughly 0.4% of total training steps for a 500B token run)
• Weight decay: $0.1$
• Gradient clipping: $1.0$
• Batch size: 4M tokens for 65B model

The shift to AdamW from Adam with standard weight decay matters primarily at this scale. In standard Adam, weight decay interacts with the second moment estimate, effectively reducing decay for parameters with large gradients. AdamW decouples the two, applying weight decay uniformly regardless of gradient history. This produces more consistent regularization.

LLaMA's decision to decay to nearly zero (final LR of $10^{-5}$, which is about 3% of the peak $3 \times 10^{-4}$) rather than the GPT convention of stopping at 10% of peak represents a more aggressive cooldown. The motivation is that the final phase of cosine decay, where the learning rate is very small, allows the model to make fine adjustments without large perturbations. In practice, the difference between decaying to 5% and 10% of peak is modest, but decaying to zero or near-zero has been consistently observed to improve final performance slightly.

A practical prior for learning rate based on observed compute-optimal configurations:

The trend is clear: optimal learning rate decreases slowly with model size in log-scale, roughly halving for every 10x increase in parameters. This relationship emerges from the combination of increased gradient sensitivity and the need for stable optimization at large scale.

The choice of weight decay $0.1$ in GPT-style training deserves explanation. Weight decay at this scale is applied to weight matrices only (not biases, not embedding tables, not layer norm parameters). At this level:

• It provides moderate regularization without strongly suppressing parameter magnitudes
• It interacts well with the cosine schedule (parameters shrink toward zero as the learning rate decays)
• It is large enough to provide a meaningful regularization signal on large models

Lower values ($0.01$) are used when the dataset is very large relative to model size (less risk of overfitting). Higher values ($0.3$) are sometimes used for smaller models or fine-tuning scenarios.

The question of which parameters receive weight decay has a principled answer that goes beyond arbitrary convention. Biases and layer normalization parameters are scalar offsets and scales that adjust the mean and variance of activations. Applying weight decay to them would bias the model toward activations with zero mean and unit variance regardless of what the data suggests, which can impede the model's ability to represent the appropriate activation statistics for a given layer. Embedding parameters represent specific token identities; shrinking them toward zero would make all token embeddings converge and lose their distinctiveness. By restricting weight decay to weight matrices, you regularize the relational structure of the model (how inputs are transformed) without constraining its representational offsets.