Predicting Model Performance › Key Parameters
Michael BrenndoerferNovember 2, 20251 min read
Language AI HandbookAll chapters Part 23 of 67Scaling Laws
Predicting Model Performance › Key ParametersOne section · Sections in this chapter9 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The parameters for scaling law prediction are:
- E (irreducible loss): The theoretical minimum loss achievable with infinite compute, determined by the entropy of natural language.
- A, B (scaling coefficients): Control how much model size and data size reduce loss respectively.
- α, β (scaling exponents): Determine how quickly returns diminish as each dimension scales (typically ~0.34 and ~0.28).
- n_samples: Number of Monte Carlo samples for uncertainty quantification. More samples provide tighter confidence intervals.
- holdout_fraction: Fraction of largest training runs to reserve for validation when testing extrapolation accuracy.