Phi Models › Phi Models

What if smaller models could match larger ones? The Phi series from Microsoft Research challenged the prevailing "scale is all you need" paradigm by showing that carefully curated training data matters as much as, or more than, model size. A 1.3 billion parameter model matching GPT-3.5 on certain benchmarks seemed implausible until Phi proved otherwise.

The Phi models represent a philosophical shift in language model development. Rather than training on all available web text and hoping scale will overcome noise, the Phi team focused obsessively on data quality. They synthesized "textbook-quality" training data using larger models, creating datasets designed to teach reasoning and coding alongside world knowledge. This approach treats model training as curriculum design rather than data accumulation.

This chapter explores the Phi model family from Phi-1 through Phi-3. You'll learn about the textbook-quality data hypothesis, understand the architectural choices that complement high-quality training, and see how small language models can achieve surprising capabilities when trained thoughtfully.