Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Phi Models › Phi-3: Quality Meets Scale

Michael BrenndoerferUpdated August 3, 20255 min read

Language AI HandbookAll chapters

Part 29 of 67Modern Decoder Models

Chapter 213Phi Models › Phi-3: Quality Meets Scale

Reading section 6 of 12 · Phi-3: Quality Meets Scale · 5 minSection 6 of 12 · 5 min

Section 6 of 12 · 5 min read

Phi Models › Phi-3: Quality Meets ScaleOne section · Sections in this chapter6 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Phi-3, released in 2024, represents the maturation of the textbook-quality approach. With 3.8B parameters in the mini variant, it was trained on an unprecedented 3.3 trillion tokens of heavily filtered and synthetic data. Phi-3 closed the gap with much larger models across nearly all benchmarks.

Data Pipeline InnovationsLink Copied

The Phi-3 training pipeline incorporated several innovations:

Multi-stage filtering: Multiple rounds of quality filtering using trained classifiers
Diverse synthetic generation: Using multiple large models (not just GPT-4) to generate diverse content
Domain coverage expansion: Systematic coverage of academic subjects, professional domains, and world knowledge
Instruction tuning: Post-training alignment using synthetic instruction-response pairs

In[21]:

Code

# Phi-3 training stages
phi3_stages = {
    "Stage 1: Web Filtering": {
        "input": "Raw web data",
        "process": "Quality classifiers, deduplication",
        "output": "~1T high-quality tokens",
    },
    "Stage 2: Synthetic Generation": {
        "input": "Topic outlines, skill maps",
        "process": "Multi-model generation, verification",
        "output": "~500B synthetic tokens",
    },
    "Stage 3: Data Mixing": {
        "input": "All filtered and synthetic data",
        "process": "Curriculum-based mixing, repetition tuning",
        "output": "3.3T training tokens",
    },
    "Stage 4: Instruction Tuning": {
        "input": "Pre-trained model",
        "process": "SFT + DPO alignment",
        "output": "Phi-3-instruct variants",
    },
}

Out[22]:

Console

Phi-3 Training Pipeline
=================================================================

Stage 1: Web Filtering
----------------------------------------
  Input: Raw web data
  Process: Quality classifiers, deduplication
  Output: ~1T high-quality tokens

Stage 2: Synthetic Generation
----------------------------------------
  Input: Topic outlines, skill maps
  Process: Multi-model generation, verification
  Output: ~500B synthetic tokens

Stage 3: Data Mixing
----------------------------------------
  Input: All filtered and synthetic data
  Process: Curriculum-based mixing, repetition tuning
  Output: 3.3T training tokens

Stage 4: Instruction Tuning
----------------------------------------
  Input: Pre-trained model
  Process: SFT + DPO alignment
  Output: Phi-3-instruct variants

Phi-3 Model VariantsLink Copied

Phi-3 comes in multiple sizes to suit different deployment scenarios:

In[23]:

Code

phi3_variants = {
    "Phi-3-mini": {
        "params": 3.8,
        "context": 128000,
        "target": "Edge devices, mobile",
    },
    "Phi-3-small": {
        "params": 7,
        "context": 128000,
        "target": "Balanced performance",
    },
    "Phi-3-medium": {
        "params": 14,
        "context": 128000,
        "target": "High performance",
    },
}

Out[24]:

Console

Phi-3 Model Variants
=======================================================
Variant          Parameters   Context      Target Use
-------------------------------------------------------
Phi-3-mini       3.8         B 128,000      Edge devices, mobile
Phi-3-small      7.0         B 128,000      Balanced performance
Phi-3-medium     14.0        B 128,000      High performance

Out[25]:

Visualization

Connected scatter plot showing the progression from Phi-1 to Phi-3-mini in terms of parameters and training tokens. — Evolution of the Phi model family. Each generation increased both model size and training data while maintaining the focus on data quality. The trajectory shows how the textbook-quality approach scaled from a code-focused 1.3B model to a general-purpose 3.8B model competing with much larger systems.

The 128K context length is notable, achieved through RoPE scaling techniques. Standard RoPE is trained for a fixed context length, but the frequencies can be adjusted to extrapolate to longer sequences. Phi-3 uses a technique where the base frequency is modified:

\theta'_i = \theta_i \cdot s = \frac{s}{\text{base}^{2i/d}}

where $s$ is a scaling factor that stretches the rotational frequencies, letting the model to handle longer sequences than it was originally trained on. This allows Phi-3 models to process long documents, complete codebases, or extended conversations while maintaining their small footprint.

Phi-3 PerformanceLink Copied

Phi-3-mini matches or exceeds much larger models on key benchmarks:

In[26]:

Code

phi3_comparison = {
    "MMLU": {
        "Phi-3-mini (3.8B)": 68.8,
        "Mixtral-8x7B (47B)": 70.6,
        "GPT-3.5-turbo": 70.0,
        "LLaMA-3-8B": 66.6,
    },
    "GSM8K": {
        "Phi-3-mini (3.8B)": 82.5,
        "Mixtral-8x7B (47B)": 74.4,
        "GPT-3.5-turbo": 57.1,
        "LLaMA-3-8B": 79.6,
    },
    "HumanEval": {
        "Phi-3-mini (3.8B)": 58.5,
        "Mixtral-8x7B (47B)": 40.2,
        "GPT-3.5-turbo": 48.1,
        "LLaMA-3-8B": 62.2,
    },
}

Out[27]:

Visualization

Grouped bar chart showing Phi-3-mini matching much larger models on MMLU, GSM8K, and HumanEval. — Phi-3-mini performance compared to larger models. Despite having only 3.8B parameters, Phi-3-mini matches GPT-3.5 on MMLU and significantly outperforms it on mathematical reasoning (GSM8K). This represents a major efficiency breakthrough.