Predicting Model Performance › Summary
Language AI HandbookAll chapters Predicting Model Performance › SummaryOne section · Sections in this chapter8 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Predicting model performance turns scaling laws from retrospective analysis into prospective planning tools:
Loss extrapolation uses fitted scaling laws to predict test loss at untrained scales, with accuracy typically within 5-10% for moderate extrapolation factors.
- Capability prediction is harder than loss prediction because task success often exhibits phase transitions rather than smooth improvement
- Uncertainty quantification through parameter sampling, holdout validation, and model comparison provides confidence bounds needed for decision-making
- Prediction reliability depends on stable training conditions, limited extrapolation distance, smoothly-improving capabilities, and maintained data quality
- Practical workflows use scaling ladders, small training runs that inform predictions before committing to expensive final training
The tension in performance prediction is that the capabilities we most want to forecast, novel emergent abilities, are those that scaling laws fail to predict. Loss extrapolation works well because loss improves smoothly; capabilities can jump discontinuously.
Yet even imperfect predictions affect research economics. Knowing that a 70B model will likely achieve 2.1 loss rather than 2.5 loss, even with 10% uncertainty, justifies or rules out billion-dollar training investments. The next chapter on emergence will examine why some capabilities resist prediction entirely, and what that implies for the future of capability forecasting.