Phi Models › Phi-1: Textbooks Are All You Need
Language AI HandbookAll chapters Phi Models › Phi-1: Textbooks Are All You NeedOne section · Sections in this chapter3 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The first Phi model focused on code generation. The team hypothesized that the quality of training data for coding tasks could be dramatically improved by generating synthetic "textbook" content rather than relying on scraped code from GitHub.
The Textbook-Quality Data Hypothesis
Real-world code has problems as training data:
- Noise: Comments may be wrong, outdated, or missing entirely
- Inconsistency: Different projects use different styles and patterns
- Complexity: Production code often handles edge cases that obscure core concepts
- Lack of explanation: Code exists without pedagogical context
A programming textbook, by contrast, introduces concepts systematically, provides clear explanations, uses consistent style, and builds complexity gradually. The Phi team used GPT-3.5 to generate synthetic textbook content covering Python programming concepts.
# Example of the difference between web-scraped vs textbook-style code
# Web-scraped style (from real repositories)
webscrape_example = """
def proc(lst, k=None):
# TODO: refactor
res = []
for i, x in enumerate(lst):
if k and i >= k:
break
if x is not None and str(x).strip():
res.append(x.lower() if hasattr(x, "lower") else x)
return res or None # legacy compat
"""
# Textbook style (synthetic, educational)
textbook_example = '''
def filter_and_normalize(items, limit=None):
"""
Filter out empty items and normalize strings to lowercase.
This function demonstrates two common list operations:
1. Filtering: removing None and empty string values
2. Normalization: converting strings to lowercase
Args:
items: A list of items to process
limit: Optional maximum number of items to return
Returns:
A list of filtered and normalized items
Example:
>>> filter_and_normalize(["Hello", None, "WORLD", ""])
["hello", "world"]
"""
result = []
for item in items:
# Skip None values
if item is None:
continue
# Skip empty strings (after stripping whitespace)
if isinstance(item, str) and not item.strip():
continue
# Normalize strings to lowercase
if isinstance(item, str):
result.append(item.lower())
else:
result.append(item)
# Stop if we have reached the limit
if limit is not None and len(result) >= limit:
break
return result
'''Web-Scraped Code vs Textbook-Quality Code
============================================================
--- Web-Scraped Style ---
def proc(lst, k=None):
# TODO: refactor
res = []
for i, x in enumerate(lst):
if k and i >= k:
break
if x is not None and str(x).strip():
res.append(x.lower() if hasattr(x, "lower") else x)
return res or None # legacy compat
--- Textbook Style ---
def filter_and_normalize(items, limit=None):
"""
Filter out empty items and normalize strings to lowercase.
This function demonstrates two common list operations:
1. Filtering: removing None and empty string values
2. Normalization: converting strings to lowercase
Args:
items: A list of items to process
limit: Optional maximum number of items to return
Returns:
A list of filtered and normalized items
Example:
filter_and_normalize(["Hello", None, "WORLD", ""])
["hello", "world"]
"""
result = []
for item in items:
# Skip None values
if item is None:
continue
# Skip empty strings (after stripping whitespace)
if isinstance(item, str) and not item.strip():
continue
# Normalize strings to lowercase
if isinstance(item, str):
result.append(item.lower())
else:
result.append(item)
# Stop if we have reached the limit
if limit is not None and len(result) >= limit:
break
return resultThe textbook-style code is longer, but every line serves an educational purpose. Variable names are descriptive. Comments explain why the code works instead of merely describing what it does. The docstring includes a concrete example. This is the kind of content that teaches programming concepts effectively.
Phi-1 Training Data Composition
Phi-1's training data combined three sources:
- Filtered Code: About 6B tokens of web code filtered for quality
- Synthetic Textbooks: 1B tokens of GPT-3.5 generated educational content
- Synthetic Exercises: Code exercises with solutions, also generated
The filtering process was aggressive. From a large corpus of Python code, only about 20% passed quality filters based on educational value metrics. The synthetic data was generated with careful prompting to produce content resembling high-quality textbooks and courses.
# Phi-1 training data breakdown
phi1_data = {
"Filtered Web Code": 6.0,
"Synthetic Textbooks": 1.0,
"Synthetic Exercises": 0.2,
}
total_tokens = sum(phi1_data.values())
Phi-1 Results
Phi-1's performance surprised the research community. On HumanEval, a benchmark for Python code generation, Phi-1 achieved 50.6% pass@1, comparable to models 10x its size.
# HumanEval benchmark comparison
humaneval_results = {
"Phi-1 (1.3B)": 50.6,
"StarCoder (15B)": 33.6,
"CodeGen-16B": 29.3,
"GPT-3.5": 48.1,
"code-davinci-002": 47.0,
}
This result validated the core hypothesis: training data quality can substitute for model scale. A small model trained on carefully curated educational content matched or exceeded much larger models trained on raw web data.