Compute-Optimal Training › Summary

This chapter translated scaling laws into practical training recipes. The key takeaways are:

• Compute-optimal allocation balances model size and training tokens according to $D \approx 20N$, where both scale as $\sqrt{C}$ with compute budget
• Planning a training run involves calculating total FLOPs, deriving optimal parameter and token counts, selecting an architecture, and sizing the data pipeline accordingly
• Training efficiency depends critically on Model FLOPs Utilization, which varies from 30-60% depending on hardware configuration, batch size, and implementation quality
• Hyperparameters scale with model size: larger models use smaller learning rates, larger batch sizes, and longer sequences
• Practical recipes vary by scale, from single-node runs at 10^21 FLOPs to distributed training at 10^24 FLOPs and beyond
• Deviation from optimality is appropriate when inference costs dominate, data is constrained, or capability thresholds matter more than raw loss

The next chapter on Data-Constrained Scaling explores what happens when you can't reach the optimal token count, examining strategies for training under data limitations and the effectiveness of data repetition.