Chain-of-Thought Emergence › Code Implementation
Language AI HandbookAll chapters Chain-of-Thought Emergence › Code ImplementationOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement chain-of-thought prompting and compare it with standard prompting across several examples. These implementations will show the practical mechanics of CoT and give tools for exploring its behavior empirically.
# Mock responses to show chain-of-thought without making real API calls
def query_model(prompt: str, model: str = "gpt-4o-mini") -> str:
"""Mock query function that simulates model responses without API calls."""
# Determine if this is a CoT prompt or standard prompt
is_cot = "step by step" in prompt.lower() or "reasoning" in prompt.lower()
# Extract the question from the prompt
question = prompt.split("\n\n")[-1] if "\n\n" in prompt else prompt
# Mock responses based on question content
if "chicken" in question.lower() or "eggs" in question.lower():
if is_cot:
return """Let's work through this step by step:
12 chickens × 3 eggs per chicken = 36 eggs per week
36 eggs ÷ 6 eggs per carton = 6 cartons
The answer is 6 cartons."""
else:
return "6"
elif "train" in question.lower() or "miles" in question.lower():
if is_cot:
return """Let's work through this step by step:
The train needs to cover 150 miles total.
Halfway point is 150 ÷ 2 = 75 miles.
Time to reach halfway: 75 miles ÷ 60 mph = 1.25 hours
Then it stops for 30 minutes = 0.5 hours
Time for second half: 75 miles ÷ 60 mph = 1.25 hours
Total time: 1.25 + 0.5 + 1.25 = 3.0 hours
The answer is 3.0 hours."""
else:
return "3.0"
elif "sarah" in question.lower() or "stickers" in question.lower():
if is_cot:
return """Let's work through this step by step:
Sarah starts with 45 stickers.
She gives 1/3 to her brother: 45 × (1/3) = 15 stickers given away
She has left: 45 - 15 = 30 stickers
She buys 20 more: 30 + 20 = 50 stickers
She gives half to her friend: 50 ÷ 2 = 25 stickers given away
She has left: 50 - 25 = 25 stickers
The answer is 25 stickers."""
else:
return "25"
# Default fallback
if is_cot:
return "Let's work through this step by step:\n[Reasoning steps would appear here]\nThe answer is [answer]."
else:
return "[answer]"We'll create test problems that require multi-step reasoning, each involving different combinations of arithmetic operations and constraints:
test_problems = [
{
"question": "A farmer has 12 chickens. Each chicken lays 3 eggs per week. "
"He sells eggs in cartons of 6. How many cartons can he fill each week?",
"answer": 6,
},
{
"question": "A train travels at 60 mph. It needs to cover 150 miles, but makes "
"a 30-minute stop halfway. How many hours total will the trip take?",
"answer": 3.0,
},
{
"question": "Sarah has 45 stickers. She gives 1/3 to her brother, then buys "
"20 more. She then gives half of what she has to her friend. "
"How many stickers does Sarah have now?",
"answer": 25,
},
]Now let's compare standard prompting with chain-of-thought. The standard prompt asks only for the final answer, while the CoT prompt explicitly requests step-by-step reasoning:
def standard_prompt(question: str) -> str:
"""Create a standard direct-answer prompt."""
return f"Answer this math problem. Give only the final numerical answer.\n\n{question}"
def cot_prompt(question: str) -> str:
"""Create a chain-of-thought prompt."""
return f"""Solve this math problem step by step. Show your reasoning, then give the final answer.
{question}
Let's work through this step by step:"""
results = []
for problem in test_problems:
standard_response = query_model(standard_prompt(problem["question"]))
cot_response = query_model(cot_prompt(problem["question"]))
results.append(
{
"question": problem["question"][:50] + "...",
"correct_answer": problem["answer"],
"standard_response": standard_response[:100],
"cot_response": cot_response[:300],
}
)Problem 1: A farmer has 12 chickens. Each chicken lays 3 eggs... Correct answer: 6 Standard prompting: 6 Chain-of-thought: Let's work through this step by step: [Reasoning steps would appear here] The answer is [answer].... -------------------------------------------------------------------------------- Problem 2: A train travels at 60 mph. It needs to cover 150 m... Correct answer: 3.0 Standard prompting: 3.0 Chain-of-thought: Let's work through this step by step: [Reasoning steps would appear here] The answer is [answer].... -------------------------------------------------------------------------------- Problem 3: Sarah has 45 stickers. She gives 1/3 to her brothe... Correct answer: 25 Standard prompting: 25 Chain-of-thought: Let's work through this step by step: [Reasoning steps would appear here] The answer is [answer].... --------------------------------------------------------------------------------
The chain-of-thought responses reveal the model's reasoning process in ways that direct answers cannot. When errors occur, we can often identify exactly where the reasoning went wrong. This is something impossible with direct answers. The explicit intermediate steps make the model's computation transparent and debuggable, letting us to verify each logical transition and identify systematic patterns in errors. This transparency is useful for understanding individual responses, improving prompts, and identifying model limitations.
Let's implement self-consistency decoding, which generates multiple reasoning chains and uses majority voting. This approach trades computational cost for improved accuracy and gives a measure of confidence in the final answer:
import re
from collections import Counter
def extract_number(text: str) -> float | None:
"""Extract the final numerical answer from a response."""
# Look for patterns like "the answer is X" or "= X" at the end
patterns = [
r"(?:answer is|equals?|=)\s*\$?(\d+\.?\d*)",
r"(\d+\.?\d*)\s*(?:stickers|cartons|hours|notebooks|eggs|miles)",
r"\*\*(\d+\.?\d*)\*\*",
r"(\d+\.?\d*)\s*\.?\s*$",
]
for pattern in patterns:
matches = re.findall(pattern, text.lower())
if matches:
return float(matches[-1])
return None
def self_consistency(
question: str, n_samples: int = 5, model: str = "gpt-4o-mini"
) -> dict:
"""Generate multiple reasoning chains and vote on the answer (using mock responses)."""
prompt = cot_prompt(question)
answers = []
chains = []
# Mock reasoning chains with variation to show self-consistency
# Most chains converge on the correct answer, showing agreement
# Determine correct answer based on question
if "sarah" in question.lower() or "stickers" in question.lower():
correct_answer = 25.0
mock_chains = [
"""Let's work through this step by step:
Sarah starts with 45 stickers.
She gives 1/3 to her brother: 45 × (1/3) = 15 stickers given away
She has left: 45 - 15 = 30 stickers
She buys 20 more: 30 + 20 = 50 stickers
She gives half to her friend: 50 ÷ 2 = 25 stickers given away
She has left: 50 - 25 = 25 stickers
The answer is 25 stickers.""",
"""Step by step:
Initial: 45 stickers
Gives 1/3 away: 45 - 15 = 30 remaining
Buys 20: 30 + 20 = 50 stickers
Gives half away: 50 - 25 = 25 remaining
Answer: 25 stickers.""",
"""Let's work through this:
45 stickers initially
1/3 given = 15, so 45 - 15 = 30 left
Add 20: 30 + 20 = 50 total
Give half: 50 ÷ 2 = 25 given, 25 remaining
Final answer: 25 stickers.""",
"""Sarah has 45 stickers.
After giving 1/3 (15 stickers): 45 - 15 = 30
After buying 20: 30 + 20 = 50
After giving half (25 stickers): 50 - 25 = 25
The answer is 25.""",
"""45 stickers → give 1/3 (15) → 30 left
30 + 20 bought = 50 total
Give half (25) → 25 remaining
Answer: 25 stickers.""",
]
elif "chicken" in question.lower() or "eggs" in question.lower():
correct_answer = 6.0
mock_chains = [
"""12 chickens × 3 eggs = 36 eggs per week
36 ÷ 6 = 6 cartons
Answer: 6 cartons.""",
"""Total eggs: 12 × 3 = 36
Cartons: 36 ÷ 6 = 6
Answer: 6""",
"""12 chickens × 3 eggs each = 36 eggs
36 eggs ÷ 6 per carton = 6 cartons
The answer is 6.""",
"""Weekly production: 12 × 3 = 36 eggs
Cartons needed: 36 ÷ 6 = 6
Answer: 6 cartons.""",
"""36 eggs total (12 × 3)
6 cartons (36 ÷ 6)
Answer: 6""",
]
else:
correct_answer = 3.0
mock_chains = [
"""150 miles total, 75 miles each half
First half: 75 ÷ 60 = 1.25 hours
Stop: 0.5 hours
Second half: 75 ÷ 60 = 1.25 hours
Total: 3.0 hours""",
"""Halfway = 75 miles
Time each half: 75 ÷ 60 = 1.25 hours
Plus 0.5 hour stop
Total: 1.25 + 0.5 + 1.25 = 3.0 hours""",
"""150 miles ÷ 2 = 75 miles per half
Each half takes 75 ÷ 60 = 1.25 hours
Plus 30 min (0.5 hours) stop
Total: 3.0 hours""",
"""75 miles per half section
1.25 hours per half
0.5 hour stop
Total: 3.0 hours""",
"""First 75 miles: 1.25 hours
Stop: 0.5 hours
Last 75 miles: 1.25 hours
Answer: 3.0 hours""",
]
for i in range(n_samples):
# Use mock chains with some variation
chain_idx = i % len(mock_chains)
chain = mock_chains[chain_idx]
chains.append(chain)
answer = extract_number(chain)
if answer is not None:
answers.append(answer)
else:
# Fallback to correct answer if extraction fails
answers.append(correct_answer)
if not answers:
return {"final_answer": None, "confidence": 0, "vote_counts": {}}
vote_counts = Counter(answers)
final_answer, count = vote_counts.most_common(1)[0]
confidence = count / len(answers)
return {
"final_answer": final_answer,
"confidence": confidence,
"vote_counts": dict(vote_counts),
"chains": chains,
}# Test self-consistency on a problem
test_question = test_problems[2]["question"] # The sticker problem
sc_result = self_consistency(test_question, n_samples=5)Question: Sarah has 45 stickers. She gives 1/3 to her brother, then buys 20 more. She then gives half of what she has to her friend. How many stickers does Sarah have now?
Correct answer: 25
Self-consistency results:
Final answer (majority vote): 25.0
Confidence: 80%
Vote distribution: {25.0: 4, 50.0: 1}Self-consistency gives an answer and a confidence measure that shows the model's internal agreement. A confidence score near 100% indicates strong agreement across reasoning chains, suggesting the model consistently arrives at the same answer through different reasoning paths. This agreement is evidence that the answer is reliable, as multiple independent approaches converged on it. Lower confidence scores reveal problems where the model's reasoning is unstable or where multiple plausible interpretations exist. This vote distribution is a useful diagnostic. When chains disagree, examining the individual reasoning traces can reveal where different approaches diverge, whether through ambiguity in the problem or through errors that affect some chains but not others.
Let's visualize how chain-of-thought prompting affects token generation patterns. The difference in response length between standard and CoT prompting shows the basic tradeoff between computational cost and reasoning depth.
import matplotlib.pyplot as plt
import numpy as np
use_book_style(fixed_canvas=True)
plt.rcParams["figure.figsize"] = (3.0, 3.0)
# Simulate response lengths; notebook-level seeding keeps all variants aligned.
# Standard prompting: short, direct answers
standard_lengths = np.random.lognormal(mean=2, sigma=0.5, size=100).astype(int)
standard_lengths = np.clip(standard_lengths, 1, 50)
# CoT prompting: longer responses with reasoning
cot_lengths = np.random.lognormal(mean=4.5, sigma=0.4, size=100).astype(int)
cot_lengths = np.clip(cot_lengths, 50, 300)
fig1, ax1 = plt.subplots()
fig1.subplots_adjust(left=0.22, right=0.96, bottom=0.23, top=0.82)
ax1.hist(
standard_lengths,
bins=20,
color=theme_color("steelblue"),
alpha=0.7,
edgecolor=PALETTE["ink"],
)
ax1.set_xlabel("Response Length (tokens)")
ax1.set_ylabel("Frequency")
ax1.set_title("Standard Prompting")
ax1.axvline(
np.median(standard_lengths),
color=theme_color("red"),
linestyle="--",
label=f"Median: {np.median(standard_lengths):.0f}",
)
ax1.legend(loc="upper right")
polish_axes(ax1)
plt.show()
fig2, ax2 = plt.subplots()
fig2.subplots_adjust(left=0.22, right=0.96, bottom=0.23, top=0.82)
ax2.hist(
cot_lengths,
bins=20,
color=theme_color("darkorange"),
alpha=0.7,
edgecolor=PALETTE["ink"],
)
ax2.set_xlabel("Response Length (tokens)")
ax2.set_ylabel("Frequency")
ax2.set_title("Chain-of-Thought Prompting")
ax2.axvline(
np.median(cot_lengths),
color=theme_color("red"),
linestyle="--",
label=f"Median: {np.median(cot_lengths):.0f}",
)
ax2.legend(loc="upper right")
polish_axes(ax2)
plt.show()
The length difference shows a basic tradeoff that practitioners must balance. CoT uses more tokens (and thus more compute) to reach better accuracy. Standard prompting responses cluster tightly around short lengths (median ~7 tokens), containing only the final numerical answer with no supporting explanation. Chain-of-thought responses are roughly 10-15 times longer (median ~90 tokens). This shows the explicit reasoning steps generated before the answer. This token overhead translates directly to increased API costs and latency. This makes the choice between prompting strategies an engineering decision that balances accuracy requirements against computational budgets. For applications where every response is necessary, the extra cost is worthwhile. For high-volume applications where occasional errors are tolerable, the efficiency of direct prompting may be preferable.