Compute-Optimal Training › Key Parameters
Michael BrenndoerferOctober 15, 20251 min read
Language AI HandbookAll chapters Part 23 of 67Scaling Laws
Compute-Optimal Training › Key ParametersOne section · Sections in this chapter10 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for compute-optimal training are:
- tokens_per_param: The optimal ratio of training tokens to model parameters, approximately 20 for Chinchilla-optimal training.
- compute_flops: Total compute budget in floating-point operations, determining the scale of training.
- mfu (Model FLOPs Utilization): Fraction of theoretical peak hardware performance achieved during training, typically 30-60%.
- peak_lr: Maximum learning rate during training, scaling inversely with model size.
- warmup_steps: Number of steps for linear learning rate warmup, typically 0.5-1% of total steps.
- batch_tokens: Number of tokens per batch, scaling with model size from 256K to 8M tokens.