Phi Models › Phi-1.5: Extending to Reasoning
Language AI HandbookAll chapters Phi Models › Phi-1.5: Extending to ReasoningOne section · Sections in this chapter4 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Building on Phi-1's success, the team created Phi-1.5 to explore whether the textbook-quality approach generalized beyond coding. Phi-1.5 maintained the 1.3B parameter size but expanded the synthetic data to cover common-sense reasoning and general knowledge.
Synthetic Data for Reasoning
Generating high-quality reasoning data is more challenging than code. Code has clear correctness criteria: it either runs or it doesn't. Reasoning depends on context and nuance, with some judgments remaining subjective. The Phi-1.5 team developed prompting strategies to generate textbook-like content for topics including:
- Science explanations
- Mathematical reasoning
- Common-sense scenarios
- Logical deduction
# Example prompt structure for generating reasoning data
reasoning_prompt_template = """
Write a short educational passage that explains the following concept
in a clear, textbook-like style. Include a concrete example and
explain the reasoning step by step.
Concept: {concept}
The passage should:
1. Introduce the concept clearly
2. Provide a relatable example
3. Walk through the reasoning explicitly
4. Summarize the key insight
"""
example_concepts = [
"Why ice floats on water",
"How compound interest works",
"Why the sky appears blue",
"The difference between correlation and causation",
]Example Concepts for Synthetic Reasoning Data ================================================== 1. Why ice floats on water 2. How compound interest works 3. Why the sky appears blue 4. The difference between correlation and causation
The resulting dataset contained approximately 20B tokens of synthetic textbook content covering diverse reasoning domains, plus 10B tokens of filtered web data selected for educational quality.
Phi-1.5 Architecture
Phi-1.5 used a standard transformer decoder architecture with some notable choices:
- 24 layers: Relatively deep for a 1.3B model
- 2048 hidden dimension: Standard width
- 32 attention heads: With and 32 heads, each head operates on dimensions
- Rotary positional embeddings (RoPE): Same as LLaMA for better length generalization
- Flash Attention: For efficient training
phi15_config = {
"Parameters": "1.3B",
"Layers": 24,
"Hidden Size": 2048,
"Attention Heads": 32,
"Head Dimension": 64,
"Vocabulary Size": 51200,
"Context Length": 2048,
"Positional Encoding": "RoPE",
"Activation": "GELU",
}Phi-1.5 Architecture Configuration ============================================= Parameters 1.3B Layers 24 Hidden Size 2048 Attention Heads 32 Head Dimension 64 Vocabulary Size 51200 Context Length 2048 Positional Encoding RoPE Activation GELU
The configuration shows a relatively deep network (24 layers) for a 1.3B parameter model, with standard choices for hidden size and attention heads. The use of RoPE positional encodings matches LLaMA, letting better length generalization than absolute positional embeddings.
The architecture itself was not novel. The innovation was entirely in the training data. This strengthens the textbook-quality hypothesis: the same architecture performs dramatically differently depending on what it's trained on.