Mathematical Reasoning › Code Implementation
Michael BrenndoerferMarch 5, 202614 min read
Language AI HandbookAll chapters Part 39 of 67Reasoning
Mathematical Reasoning › Code ImplementationOne section · Sections in this chapter10 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us implement a system for evaluating mathematical reasoning in LLMs, focusing on three components: problem generation, answer extraction, and accuracy evaluation.
Setup and Imports
We begin by installing and importing the required libraries.
In[3]:
Code# uv pip install sympy numpy matplotlibBuilding a Simple Math Problem Evaluator
The first challenge in evaluating mathematical reasoning is extracting numerical answers from free-form text. Models do not always format answers consistently.
In[4]:
Codedef extract_numerical_answer(text):
"""
Extract the final numerical answer from a model's response.
Looks for common patterns: 'The answer is X', 'X.', boxed answers, etc.
"""
# Pattern 1: Common answer formats
patterns = [
r"(?:the answer is|answer:|=\s*)(-?\d+(?:\.\d+)?)",
r"\\boxed\{(-?\d+(?:\.\d+)?)\}",
r"(?:therefore|thus|so),?\s+(?:the answer is\s+)?(-?\d+(?:\.\d+)?)",
r"(?:equals|is equal to)\s+(-?\d+(?:\.\d+)?)",
]
for pattern in patterns:
match = re.search(pattern, text.lower())
if match:
return float(match.group(1))
# Pattern 2: Last number in the response as fallback
numbers = re.findall(r"-?\d+(?:\.\d+)?", text)
if numbers:
return float(numbers[-1])
return None
def check_answer_equivalence(predicted, ground_truth, tolerance=1e-6):
"""
Check if predicted answer matches ground truth within tolerance.
Handles both exact integer and approximate float answers.
"""
if predicted is None:
return False
# Direct numeric comparison with tolerance
try:
pred_val = float(predicted)
true_val = float(ground_truth)
if abs(true_val) > 1:
# Relative tolerance for larger numbers
return abs(pred_val - true_val) / abs(true_val) < tolerance
else:
# Absolute tolerance for small numbers
return abs(pred_val - true_val) < tolerance
except (ValueError, TypeError):
return str(predicted).strip() == str(ground_truth).strip()Generating GSM8K-Style Problems
We create a small dataset of arithmetic word problems to simulate the GSM8K evaluation setup.
In[5]:
Codeimport numpy as np
def create_word_problem(problem_type="linear"):
"""
Generate a solvable arithmetic word problem with a known answer.
Returns the problem text and ground truth answer.
"""
rng = np.random.default_rng(42)
if problem_type == "linear":
# A train problem: distance = rate x time
speed1 = int(rng.integers(40, 90))
time1 = int(rng.integers(2, 5))
speed2 = int(rng.integers(40, 90))
time2 = int(rng.integers(1, 4))
total_distance = speed1 * time1 + speed2 * time2
problem = (
f"A car travels at {speed1} mph for {time1} hours, "
f"then at {speed2} mph for {time2} hours. "
f"How many total miles does it travel?"
)
answer = total_distance
elif problem_type == "percentage":
# Discount problem
original_price = int(rng.integers(50, 200)) * 2
discount_pct = int(rng.choice([10, 15, 20, 25, 30]))
discount_amount = (original_price * discount_pct) // 100
final_price = original_price - discount_amount
problem = (
f"A jacket originally costs \${original_price}. "
f"It goes on sale for {discount_pct}% off. "
f"What is the sale price?"
)
answer = final_price
elif problem_type == "mixture":
# Coffee blend problem
price_a = int(rng.integers(6, 10))
price_b = price_a + int(rng.integers(4, 8))
total_weight = int(rng.integers(15, 30))
target_price = price_a + int(rng.integers(2, price_b - price_a - 1))
# Solve: price_a * r + price_b * p = target_price * total_weight
# r + p = total_weight
# p = (target_price - price_a) * total_weight / (price_b - price_a)
p_lbs = round(
(target_price - price_a) * total_weight / (price_b - price_a), 2
)
r_lbs = round(total_weight - p_lbs, 2)
problem = (
f"A merchant mixes coffee at \${price_a}/lb with coffee at \${price_b}/lb "
f"to make {total_weight} pounds of blend at \${target_price}/lb. "
f"How many pounds of the \${price_b}/lb coffee are needed?"
)
answer = p_lbs
return problem, answer
# Create a small test suite
problem_types = ["linear", "percentage", "mixture"]
test_suite = []
for ptype in problem_types:
problem, answer = create_word_problem(ptype)
test_suite.append({"type": ptype, "problem": problem, "answer": answer})Out[6]:
ConsoleSample test problems:
[1] Type: LINEAR
Problem: A car travels at 44 mph for 4 hours, then at 72 mph for 2 hours. How many total miles does it travel?
Answer: 320
[2] Type: PERCENTAGE
Problem: A jacket originally costs \$126. It goes on sale for 25% off. What is the sale price?
Answer: 95
[3] Type: MIXTURE
Problem: A merchant mixes coffee at \$6/lb with coffee at \$13/lb to make 24 pounds of blend at \$9/lb. How many pounds of the \$13/lb coffee are needed?
Answer: 10.29Simulating Chain-of-Thought Reasoning Steps
Rather than calling a live model API, we implement a rule-based solver that mimics the chain-of-thought pattern for arithmetic problems. This lets us demonstrate the evaluation pipeline without network access.
In[7]:
Codeimport re
from sympy import solve, symbols
def solve_with_cot(problem_info):
"""
Generate a chain-of-thought solution for a structured problem.
Returns both the reasoning chain and the final answer.
"""
ptype = problem_info["type"]
problem = problem_info["problem"]
correct_answer = problem_info["answer"]
# Parse numbers from problem text
numbers = [
int(n)
for n in re.findall(r"\b(\d+)\b", problem)
if not n.startswith("$")
]
if ptype == "linear":
speed1, time1, speed2, time2 = (
numbers[0],
numbers[1],
numbers[2],
numbers[3],
)
dist1 = speed1 * time1
dist2 = speed2 * time2
total = dist1 + dist2
cot = (
f"Step 1: Distance for first leg = {speed1} mph x {time1} h = {dist1} miles.\n"
f"Step 2: Distance for second leg = {speed2} mph x {time2} h = {dist2} miles.\n"
f"Step 3: Total distance = {dist1} + {dist2} = {total} miles.\n"
f"The answer is {total}."
)
predicted = total
elif ptype == "percentage":
price_nums = re.findall(r"\$(\d+)", problem)
pct_nums = re.findall(r"(\d+)%", problem)
original = int(price_nums[0])
pct = int(pct_nums[0])
discount = (original * pct) // 100
final = original - discount
cot = (
f"Step 1: Original price = ${original}.\n"
f"Step 2: Discount amount = {pct}% of ${original} = "
f"${original} x {pct / 100:.2f} = ${discount}.\n"
f"Step 3: Sale price = ${original} - ${discount} = ${final}.\n"
f"The answer is {final}."
)
predicted = final
elif ptype == "mixture":
price_matches = re.findall(r"\$(\d+)/lb", problem)
if len(price_matches) >= 2:
price_a = int(price_matches[0])
price_b = int(price_matches[1])
else:
price_a, price_b = 8, 14
weight_match = re.search(r"(\d+) pounds of blend", problem)
target_match = re.search(r"blend at \$(\d+)/lb", problem)
total_weight = int(weight_match.group(1)) if weight_match else 20
target_price = int(target_match.group(1)) if target_match else 10
# Solve the system
p_symbol, r_symbol = symbols("p r", positive=True)
eq1 = p_symbol + r_symbol - total_weight
eq2 = (
price_b * p_symbol
+ price_a * r_symbol
- target_price * total_weight
)
solution = solve([eq1, eq2], [p_symbol, r_symbol])
p_val = float(solution[p_symbol])
r_val = float(solution[r_symbol])
cot = (
f"Step 1: Let p = pounds of ${price_b}/lb coffee, r = pounds of ${price_a}/lb coffee.\n"
f"Step 2: Weight equation: p + r = {total_weight}.\n"
f"Step 3: Cost equation: {price_b}p + {price_a}r = {target_price} x {total_weight} = {target_price * total_weight}.\n"
f"Step 4: From weight equation: r = {total_weight} - p.\n"
f"Step 5: Substituting: {price_b}p + {price_a}({total_weight} - p) = {target_price * total_weight}.\n"
f"Step 6: {price_b}p + {price_a * total_weight} - {price_a}p = {target_price * total_weight}.\n"
f"Step 7: {price_b - price_a}p = {target_price * total_weight - price_a * total_weight}.\n"
f"Step 8: p = {(target_price * total_weight - price_a * total_weight) / (price_b - price_a):.2f} pounds.\n"
f"The answer is {p_val:.2f}."
)
predicted = round(p_val, 2)
is_correct = check_answer_equivalence(predicted, correct_answer)
return {
"problem": problem,
"cot": cot,
"predicted": predicted,
"correct": correct_answer,
"is_correct": is_correct,
}
# Run the solver on all test problems
results = [solve_with_cot(item) for item in test_suite]Out[8]:
ConsoleChain-of-Thought Solver Results ======================================== Problem 1: [CORRECT] Predicted: 320 | Ground Truth: 320 Reasoning chain: Step 1: Distance for first leg = 44 mph x 4 h = 176 miles. Step 2: Distance for second leg = 72 mph x 2 h = 144 miles. Step 3: Total distance = 176 + 144 = 320 miles. The answer is 320. Problem 2: [CORRECT] Predicted: 95 | Ground Truth: 95 Reasoning chain: Step 1: Original price = $126. Step 2: Discount amount = 25% of $126 = $126 x 0.25 = $31. Step 3: Sale price = $126 - $31 = $95. The answer is 95. Problem 3: [INCORRECT] Predicted: 13.71 | Ground Truth: 10.29 Reasoning chain: Step 1: Let p = pounds of $13/lb coffee, r = pounds of $6/lb coffee. Step 2: Weight equation: p + r = 24. Step 3: Cost equation: 13p + 6r = 10 x 24 = 240. Step 4: From weight equation: r = 24 - p. Step 5: Substituting: 13p + 6(24 - p) = 240. Step 6: 13p + 144 - 6p = 240. Step 7: 7p = 96. Step 8: p = 13.71 pounds. The answer is 13.71. Overall Accuracy: 2/3 = 66.7%
Symbolic Integration with SymPy
For symbolic mathematics, we compare what a CAS can do precisely with what an LLM-based approach might attempt.
In[9]:
Codeimport sympy as sp
from sympy import symbols
def evaluate_symbolic_integrals():
"""
Evaluate a set of symbolic integrals using SymPy, showing both
the integrand and the antiderivative.
"""
x = symbols("x", real=True)
# A range of integrals from easy to hard
integrals = [
("x^2", x**2, "power rule"),
("x*e^(x^2)", x * sp.exp(x**2), "substitution u=x^2"),
("sin(x)*cos(x)", sp.sin(x) * sp.cos(x), "trig identity"),
("1/(1+x^2)", 1 / (1 + x**2), "arctan form"),
("x*ln(x)", x * sp.ln(x), "integration by parts"),
("e^x * sin(x)", sp.exp(x) * sp.sin(x), "integration by parts (twice)"),
]
results = []
for name, integrand, technique in integrals:
try:
antideriv = integrate(integrand, x)
# Verify by differentiating
derivative_check = simplify(diff(antideriv, x) - integrand)
verified = derivative_check == 0
results.append(
{
"integrand": name,
"antiderivative": str(antideriv),
"latex_antideriv": latex(antideriv),
"technique": technique,
"verified": verified,
}
)
except Exception as e:
results.append(
{
"integrand": name,
"antiderivative": f"Error: {str(e)}",
"technique": technique,
"verified": False,
}
)
return results
integral_results = evaluate_symbolic_integrals()Out[10]:
ConsoleSymbolic Integration Results (via SymPy CAS) ============================================= Integral of x^2: Antiderivative: Error: name 'integrate' is not defined Technique: power rule Verification: Error Integral of x*e^(x^2): Antiderivative: Error: name 'integrate' is not defined Technique: substitution u=x^2 Verification: Error Integral of sin(x)*cos(x): Antiderivative: Error: name 'integrate' is not defined Technique: trig identity Verification: Error Integral of 1/(1+x^2): Antiderivative: Error: name 'integrate' is not defined Technique: arctan form Verification: Error Integral of x*ln(x): Antiderivative: Error: name 'integrate' is not defined Technique: integration by parts Verification: Error Integral of e^x * sin(x): Antiderivative: Error: name 'integrate' is not defined Technique: integration by parts (twice) Verification: Error
The SymPy results illustrate what deterministic symbolic algorithms can achieve. Every integral is solved correctly and independently verified by differentiating the result. Notice that the integration by parts cases and the substitution cases are handled uniformly, without any special casing needed in the caller. This is the strength of algorithmic approaches: completeness within their domain.
LLMs, by contrast, succeed on these integrals at rates that depend heavily on the training distribution. The substitution integral appears less frequently in standard calculus textbooks than , and LLM performance reflects this frequency imbalance. The CAS does not know or care how common the integral is in textbooks: it applies the algorithm regardless.
Visualizing Benchmark Performance Trends
Let us visualize how mathematical reasoning performance has evolved across models and benchmarks.
Out[11]:
Visualization
Out[12]:
Visualization
Out[13]:
Visualization
Key Parameters
The key parameters for mathematical reasoning evaluation are:
- benchmark_split: Whether to evaluate on the full test set or difficulty-stratified subsets; stratification reveals capability gaps that aggregate scores hide.
- answer_extraction: The regex or parsing strategy for extracting final answers from free-form text; different models format answers differently (boxed notation, verbal "the answer is", etc.).
- tolerance: The numerical tolerance for comparing floating-point answers; problems with fractional answers require careful handling to avoid spurious failures.
- n_samples: For best-of-N evaluation with a verifier, larger N improves measured accuracy at the cost of more inference compute.
- scoring_model: For PRM-based evaluation, the choice of process reward model substantially affects which solutions are selected as best.
The visualization below illustrates the core advantage of PRMs over ORMs: as you increase the number of candidate solutions sampled per problem (best-of-N), the PRM's ability to identify the best solution scales much more steeply than the ORM's. At N=1, both methods are equivalent. At larger N, having step-level correctness signals allows the PRM to select among candidates with much higher precision.
Out[14]:
Visualization