Phi Models › The Small Model Challenge
Language AI HandbookAll chapters Phi Models › The Small Model ChallengeOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before Phi, the path to better language models seemed clear: make them bigger. GPT-2 (1.5B parameters) improved on GPT (117M). GPT-3 (175B) improved on GPT-2. Each generation scaled up by roughly 10x, and performance followed. This pattern suggested that capability and size were fundamentally linked.
Scaling laws describe the predictable relationship between model size, training data, and compute on one hand, and model performance on the other. Research by Kaplan et al. (2020) and Hoffmann et al. (2022) showed that loss decreases as a power law:
where:
- : the model's loss (lower is better)
- : the number of model parameters
- : the number of training tokens
- : empirically determined exponents (typically around 0.05-0.1)
This relationship suggests that larger models are reliably better given sufficient data and compute.
The Phi models consistently achieve lower loss than standard scaling laws predict. This shows that data quality can substitute for model size.
But scaling has costs. Training GPT-4 reportedly cost over $100 million. Running inference on 175B parameter models requires specialized hardware. The environmental footprint of training and deploying such models is substantial. If smaller models could achieve similar capabilities, the practical benefits would be enormous.
The Phi project asked a contrarian question: what if training data quality, rather than sheer quantity, is the primary bottleneck? Web-scraped datasets contain enormous amounts of low-quality content: SEO spam, machine-generated text, forum noise, and redundant information. What if we replaced this with carefully crafted educational content?
# Compare model sizes in the Phi family vs contemporaries
phi_models = {
"Phi-1 (2023)": {"params": 1.3, "training_tokens": 7, "focus": "Code"},
"Phi-1.5 (2023)": {
"params": 1.3,
"training_tokens": 30,
"focus": "Reasoning",
},
"Phi-2 (2023)": {"params": 2.7, "training_tokens": 250, "focus": "General"},
"Phi-3-mini (2024)": {
"params": 3.8,
"training_tokens": 3300,
"focus": "General",
},
}
comparison_models = {
"GPT-3.5": {"params": 175, "training_tokens": 300, "focus": "General"},
"LLaMA-2-7B": {"params": 7, "training_tokens": 2000, "focus": "General"},
"Mistral-7B": {"params": 7, "training_tokens": 8000, "focus": "General"},
"LLaMA-2-70B": {"params": 70, "training_tokens": 2000, "focus": "General"},
}Phi Model Family ================================================================= Model Params (B) Tokens (B) Focus ----------------------------------------------------------------- Phi-1 (2023) 1.3 7 Code Phi-1.5 (2023) 1.3 30 Reasoning Phi-2 (2023) 2.7 250 General Phi-3-mini (2024) 3.8 3300 General Comparison Models ----------------------------------------------------------------- GPT-3.5 175.0 300 General LLaMA-2-7B 7.0 2000 General Mistral-7B 7.0 8000 General LLaMA-2-70B 70.0 2000 General
The table reveals the dramatic size difference between Phi models and their competitors. Phi-1 uses only 7B training tokens compared to LLaMA-2's 2,000B, yet achieves competitive performance. This 280x difference in training data volume shows how strongly data quality affects the result.
The parameter counts tell a striking story. Phi-1 with 1.3B parameters competed with models 10-100x larger on coding tasks. Phi-2 at 2.7B matched LLaMA-2-7B and approached LLaMA-2-70B on reasoning benchmarks. How is this possible?