Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Phi Models › Phi-2: Scaling Quality

Michael BrenndoerferUpdated August 3, 20254 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 213Phi Models › Phi-2: Scaling Quality

Reading section 5 of 12 · Phi-2: Scaling Quality · 4 minSection 5 of 12 · 4 min

Section 5 of 12 · 4 min read

Phi Models › Phi-2: Scaling QualityOne section · Sections in this chapter5 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Phi-2 doubled the parameter count to 2.7B and scaled up the synthetic data pipeline. The model demonstrated that the textbook-quality approach continues to provide benefits at larger scales, maintaining competitive performance against models 10-25x its size.

Training Data StrategyLink Copied

Phi-2's training combined multiple data sources:

Synthetic textbooks: Expanded coverage of STEM topics and coding, together with reasoning
Web data: Heavily filtered for educational content using NLP classifiers
Code: High-quality repositories with documentation

The total training corpus was approximately 250B tokens, still modest compared to LLaMA-2's 2T tokens or GPT-3's 300B tokens, but dramatically higher quality according to Microsoft's metrics.

In[16]:

Code

# Training efficiency comparison (comparable-size models: ≤7B params + Mistral)
training_efficiency = {
    "Phi-2": {"params": 2.7, "tokens": 250, "tokens_per_param": 92.6},
    "LLaMA-2-7B": {"params": 7, "tokens": 2000, "tokens_per_param": 285.7},
    "LLaMA-2-13B": {"params": 13, "tokens": 2000, "tokens_per_param": 153.8},
    "Mistral-7B": {"params": 7, "tokens": 8000, "tokens_per_param": 1142.9},
}

Out[17]:

Visualization

Bar chart comparing tokens-per-parameter ratio across different models. — Training tokens per parameter across models. Phi-2 uses far fewer tokens per parameter than other models, suggesting its performance gains come from data quality rather than quantity. Mistral-7B represents the opposite extreme, training on massive token counts.

Benchmark PerformanceLink Copied

Phi-2 achieved remarkable results across diverse benchmarks, often matching or exceeding models with 7-25x more parameters.

In[18]:

Code

# Multi-benchmark comparison
benchmarks = {
    "MMLU": {
        "Phi-2 (2.7B)": 56.3,
        "Mistral-7B": 60.1,
        "LLaMA-2-7B": 45.3,
        "LLaMA-2-13B": 54.8,
        "LLaMA-2-70B": 68.9,
    },
    "GSM8K (Math)": {
        "Phi-2 (2.7B)": 57.2,
        "Mistral-7B": 37.8,
        "LLaMA-2-7B": 14.6,
        "LLaMA-2-13B": 28.7,
        "LLaMA-2-70B": 56.8,
    },
    "HumanEval (Code)": {
        "Phi-2 (2.7B)": 59.0,
        "Mistral-7B": 30.5,
        "LLaMA-2-7B": 12.8,
        "LLaMA-2-13B": 18.3,
        "LLaMA-2-70B": 29.9,
    },
    "MBPP (Code)": {
        "Phi-2 (2.7B)": 60.6,
        "Mistral-7B": 47.5,
        "LLaMA-2-7B": 20.8,
        "LLaMA-2-13B": 30.8,
        "LLaMA-2-70B": 49.8,
    },
}

Out[19]:

Visualization

Grouped bar chart comparing Phi-2 against larger models on MMLU, GSM8K, HumanEval, and MBPP benchmarks. — Phi-2 performance across multiple benchmarks compared to larger models. On GSM8K (mathematical reasoning) and code benchmarks, Phi-2 matches or exceeds LLaMA-2-70B despite being 25x smaller. This demonstrates the particular strength of textbook-quality training for reasoning tasks.

The results reveal an interesting pattern. Phi-2 excels particularly on GSM8K (mathematical reasoning) and code generation, precisely the domains where textbook-quality data provides the clearest advantage. On MMLU (general knowledge), larger models maintain an edge, suggesting that some types of knowledge benefit more from scale than from data quality.

Out[20]:

Visualization

Bar chart showing performance per billion parameters, with Phi-2 dramatically outperforming larger models on efficiency. — Performance per billion parameters on GSM8K (mathematical reasoning). Phi-2 achieves over 21 points per billion parameters, far exceeding the efficiency of larger models. This metric highlights how textbook-quality data enables more effective use of model capacity for reasoning tasks.