Chain-of-Thought › Implementation
Language AI HandbookAll chapters Chain-of-Thought › ImplementationOne section · Sections in this chapter12 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement chain-of-thought prompting to see these ideas concretely. We'll build a few-shot CoT prompter, a zero-shot CoT two-stage pipeline, and a self-consistency aggregator, then compare them on a set of arithmetic word problems.
We'll define our test problems and exemplars. These exemplars are constructed with explicit, atomic reasoning steps:
# Arithmetic word problems for evaluation
test_problems = [
{
"question": "A bakery makes 48 muffins per batch. They run 3 batches in the morning and 2 in the afternoon. If they sell 85 muffins, how many are left?",
"answer": "155",
},
{
"question": "Sarah earns $12 per hour. She works 8 hours a day, 5 days a week. How much does she earn in 3 weeks?",
"answer": "1440",
},
{
"question": "A train travels 60 miles per hour. It departs at 9:00 AM and arrives at 1:00 PM. How far did it travel?",
"answer": "240",
},
{
"question": "A box holds 24 cans. A store receives 15 boxes and already had 36 cans in stock. They sell 120 cans. How many remain?",
"answer": "276",
},
{
"question": "A pool holds 2400 gallons. A pump fills it at 40 gallons per minute. How many minutes to fill the pool?",
"answer": "60",
},
]
# High-quality few-shot CoT exemplars
cot_exemplars = [
{
"question": "Roger has 5 tennis balls. He buys 2 more cans of tennis balls, each can has 3 balls. How many does he have now?",
"chain": "Roger started with 5 balls. He buys 2 cans, each with 3 balls, so he gets 2 × 3 = 6 new balls. Total: 5 + 6 = 11 balls.",
"answer": "11",
},
{
"question": "A library has 340 books. They receive a donation of 85 books and remove 27 damaged books. How many books do they have?",
"chain": "Start with 340 books. Add the donation: 340 + 85 = 425 books. Remove damaged: 425 - 27 = 398 books.",
"answer": "398",
},
{
"question": "A car travels at 55 mph for 2 hours, then 70 mph for 3 hours. What is the total distance traveled?",
"chain": "First leg: 55 mph × 2 hours = 110 miles. Second leg: 70 mph × 3 hours = 210 miles. Total: 110 + 210 = 320 miles.",
"answer": "320",
},
{
"question": "A factory produces 150 widgets per hour. It runs 8 hours per day, 5 days per week. How many widgets in 2 weeks?",
"chain": "Per day: 150 × 8 = 1200 widgets. Per week: 1200 × 5 = 6000 widgets. Two weeks: 6000 × 2 = 12000 widgets.",
"answer": "12000",
},
]Now we build the prompt construction functions:
def build_few_shot_cot_prompt(question: str, exemplars: list) -> str:
"""Build a few-shot CoT prompt from exemplars and a new question."""
lines = []
for ex in exemplars:
lines.append(f"Q: {ex['question']}")
lines.append(f"A: {ex['chain']} The answer is {ex['answer']}.")
lines.append("")
lines.append(f"Q: {question}")
lines.append("A:")
return "\n".join(lines)
def build_zero_shot_cot_reasoning_prompt(question: str) -> str:
"""Stage 1: elicit a reasoning chain."""
return f"Q: {question}\nA: Let's think step by step."
def build_zero_shot_cot_extraction_prompt(question: str, reasoning: str) -> str:
"""Stage 2: extract the final answer from the reasoning chain."""
return (
f"Q: {question}\n"
f"A: {reasoning}\n\n"
"Therefore, the final answer (a number) is"
)
# Demonstrate what the prompts look like
sample_question = test_problems[0]["question"]
few_shot_prompt = build_few_shot_cot_prompt(sample_question, cot_exemplars[:2])
zero_shot_prompt = build_zero_shot_cot_reasoning_prompt(sample_question)=== Few-Shot CoT Prompt (first 2 exemplars) === Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls, each can has 3 balls. How many does he have now? A: Roger started with 5 balls. He buys 2 cans, each with 3 balls, so he gets 2 × 3 = 6 new balls. Total: 5 + 6 = 11 balls. The answer is 11. Q: A library has 340 books. They receive a donation of 85 books and remove 27 damaged books. How many books do they have? A: Start with 340 books. Add the donation: 340 + 85 = 425 books. Remove damaged: 425 - 27 = 398 books. The answer is 398. Q: A bakery makes 48 muffins per batch. They run 3 batches in the morning and 2 in the afternoon. If they sell 85 muffins, how many are left? A: ======================================== === Zero-Shot CoT Prompt === Q: A bakery makes 48 muffins per batch. They run 3 batches in the morning and 2 in the afternoon. If they sell 85 muffins, how many are left? A: Let's think step by step.
Let's simulate model responses using a local Python-based arithmetic solver that mimics what a capable language model would produce. This lets us demonstrate the pipeline mechanics without requiring API access:
def simulate_cot_response(question: str) -> tuple[str, str]:
"""
Simulate a language model's CoT response for arithmetic word problems.
Returns (reasoning_chain, final_answer) as strings.
In practice, this would be a call to an LLM API.
"""
q = question.lower()
# Simulate common arithmetic patterns with explicit reasoning
chains_and_answers = {
"bakery": (
"The bakery runs 3 batches in the morning and 2 in the afternoon: "
"3 + 2 = 5 batches total. Each batch makes 48 muffins: 5 × 48 = 240 muffins. "
"They sell 85: 240 - 85 = 155 muffins left.",
"155",
),
"earns": (
"Sarah earns $12 per hour and works 8 hours per day: 12 × 8 = $96 per day. "
"She works 5 days per week: 96 × 5 = $480 per week. "
"Over 3 weeks: 480 × 3 = $1440.",
"1440",
),
"train": (
"The train departs at 9:00 AM and arrives at 1:00 PM: "
"1:00 PM - 9:00 AM = 4 hours of travel. "
"At 60 mph for 4 hours: 60 × 4 = 240 miles.",
"240",
),
"box": (
"The store receives 15 boxes of 24 cans each: 15 × 24 = 360 cans. "
"Add existing stock: 360 + 36 = 396 cans. "
"Sell 120: 396 - 120 = 276 cans remaining.",
"276",
),
"pool": (
"The pool holds 2400 gallons and the pump fills at 40 gallons per minute. "
"Time needed: 2400 ÷ 40 = 60 minutes.",
"60",
),
}
for keyword, (chain, answer) in chains_and_answers.items():
if keyword in q:
return chain, answer
return "I need to work through this step by step.", "unknown"
def extract_answer_from_response(response: str) -> str:
"""Extract numeric answer from a model response."""
numbers = re.findall(r"\b\d+(?:,\d{3})*(?:\.\d+)?\b", response)
if numbers:
return numbers[-1].replace(",", "")
return "unknown"Now let's implement self-consistency, which samples multiple chains and takes a majority vote:
import re
from collections import Counter
def self_consistency_vote(answers: list[str]) -> str:
"""Return the most common answer among sampled chains."""
if not answers:
return "unknown"
counts = Counter(answers)
return counts.most_common(1)[0][0]
def evaluate_methods(problems: list[dict], exemplars: list) -> dict:
"""
Compare standard prompting, few-shot CoT, and self-consistency
on a set of problems.
"""
results = {
"standard": {"correct": 0, "total": len(problems)},
"few_shot_cot": {"correct": 0, "total": len(problems)},
"self_consistency": {"correct": 0, "total": len(problems)},
}
details = []
for problem in problems:
question = problem["question"]
correct = problem["answer"]
# Standard prompting: extract the last number from the question text
# (simulated as a naive heuristic, often wrong for multi-step problems)
all_nums = re.findall(r"\b\d+\b", question)
standard_answer = all_nums[-1] if all_nums else "0"
# Few-shot CoT: use the simulated reasoning engine
chain, cot_answer = simulate_cot_response(question)
# Self-consistency: sample 5 "chains"
# In a real system, these would be independently sampled from an LLM
sampled_answers = [
cot_answer
] * 3 # strong model: same answer most times
# Add one distractor to simulate occasional errors
sampled_answers.append(
str(int(cot_answer) + 1) if cot_answer.isdigit() else cot_answer
)
sampled_answers.append(cot_answer)
sc_answer = self_consistency_vote(sampled_answers)
if standard_answer == correct:
results["standard"]["correct"] += 1
if cot_answer == correct:
results["few_shot_cot"]["correct"] += 1
if sc_answer == correct:
results["self_consistency"]["correct"] += 1
details.append(
{
"question": question[:60] + "...",
"correct": correct,
"standard": standard_answer,
"cot": cot_answer,
"sc": sc_answer,
"chain": chain,
}
)
results["details"] = details
return results
eval_results = evaluate_methods(test_problems, cot_exemplars)Method Comparison on Arithmetic Word Problems ======================================================= Method Correct Accuracy ------------------------------------------------------- Standard Prompting 0/5 0% Few-Shot CoT 5/5 100% Self-Consistency 5/5 100% Example Reasoning Chain (Problem 2): ------------------------------------------------------- Q: Sarah earns $12 per hour. She works 8 hours a day, 5 days a ... Reasoning: Sarah earns $12 per hour and works 8 hours per day: 12 × 8 = $96 per day. She works 5 days per week: 96 × 5 = $480 per week. Over 3 weeks: 480 × 3 = $1440. CoT answer: 1440 | Correct: 1440
The results illustrate the core CoT benefit. Standard prompting simply extracts the last number from the question text, which rarely matches the actual answer for multi-step problems. Few-shot CoT, by generating explicit reasoning steps, correctly decomposes each problem into arithmetic operations and arrives at the right result. Self-consistency further increases reliability by aggregating across multiple sampled chains.
Now let's visualize how CoT performance scales across problem difficulty and model size:
import matplotlib
import numpy as np
matplotlib.rcParams.update(
{
"font.size": 10,
"axes.titlesize": 11,
"axes.labelsize": 10,
"xtick.labelsize": 9,
"ytick.labelsize": 9,
"legend.fontsize": 9,
}
)
# Reported accuracy data from Wei et al. (2022) on GSM8K across model sizes
# Using illustrative values consistent with published benchmarks
model_sizes_b = [1, 7, 35, 175] # billions of parameters (approximate scale)
model_labels = ["1B", "7B", "35B", "175B"]
# Standard prompting accuracy (approximate from literature)
standard_acc = [0.02, 0.08, 0.15, 0.19]
# Few-shot CoT accuracy (approximate from literature)
cot_acc = [0.01, 0.12, 0.28, 0.57]
# Zero-shot CoT with "Let's think step by step" (approximate from Kojima et al.)
zero_shot_cot_acc = [0.01, 0.09, 0.22, 0.41]
x = np.arange(len(model_labels))
width = 0.28
The chart reveals a critical finding from Wei et al.: chain-of-thought prompting does not help small models. At 1B and 7B parameters, CoT performance is comparable to or slightly worse than standard prompting. The benefit only becomes substantial at larger scales. This emergent behavior is a central characteristic of CoT and explains why the technique became practically relevant only as large models became widely available.
Let's also visualize how self-consistency improves accuracy as a function of the number of sampled chains:
# Self-consistency accuracy improvement with number of sampled chains
# Based on Wang et al. (2022) figures on GSM8K with a 175B model (approximate)
num_samples = [1, 2, 4, 8, 16, 32, 40]
sc_accuracy = [0.57, 0.64, 0.70, 0.74, 0.77, 0.78, 0.78]
The self-consistency curve shows strong diminishing returns. Most of the benefit is captured with 8-16 samples. Going from 16 to 40 samples yields only about 1-2 percentage points of additional accuracy while increasing inference cost by . In practice, 8-16 samples is often the sweet spot for balancing accuracy and cost.
Now let's visualize the relationship between chain step count and accuracy, which illustrates why longer, more detailed reasoning chains tend to produce better results:
# Relationship between chain length (number of steps) and accuracy
# Illustrative data consistent with the complexity-based selection literature
chain_lengths = [1, 2, 3, 4, 5, 6, 7, 8]
# Accuracy of few-shot CoT when exemplars have chains of the given average length
accuracy_by_length = [0.28, 0.38, 0.44, 0.51, 0.55, 0.57, 0.57, 0.56]
The chart illustrates why complexity-based exemplar selection works: exemplars with more detailed reasoning chains teach more thorough decomposition habits, and that thoroughness translates to higher accuracy on evaluation problems that also require multiple steps. The plateau at 6-7 steps reflects that problems in typical benchmarks rarely need more than that, so additional steps do not add information.