Mathematical Reasoning › Math Benchmarks in Detail

Benchmarking mathematical reasoning requires careful choices about what to measure. Several benchmarks have become standard references, each with specific strengths and known weaknesses.

GSM8K (Grade School Math) contains 8,500 diverse grade school math word problems. Each problem requires 2 to 8 reasoning steps using addition, subtraction, multiplication, and division. The problems are linguistically diverse and designed to avoid simple pattern matching.

A key feature of GSM8K is that solutions are annotated with step-by-step reasoning, making it suitable for both evaluation and chain-of-thought training. The test set of 1,319 problems is the standard evaluation split. The problems were written by human contractors who were instructed to create problems that could not be solved by keyword matching or simple heuristics, which is why GSM8K proved to be a meaningful benchmark even though the underlying arithmetic is elementary.

Performance on GSM8K has been used as a proxy for mathematical capability across model families. The progression from GPT-3 (around 35 percent with few-shot chain-of-thought) to GPT-4 (over 90 percent) traces the rapid improvement in mathematical reasoning over 2021 to 2024. Today, GSM8K performance is approaching saturation for frontier models, and new benchmarks are needed to differentiate capability.

One known limitation of GSM8K is contamination risk. Because the benchmark was released publicly, its problems have appeared across the internet in blog posts, tutorials, and fine-tuning datasets. Models trained after the benchmark's release may have seen the exact test problems, inflating apparent performance. Researchers constructing new evaluations now routinely include "canary" problems designed to detect memorization, or construct problems programmatically to ensure they cannot appear in training data.

The MATH benchmark contains 12,500 problems from high school mathematics competitions, organized into seven subject areas:

• Pre-Algebra
• Algebra
• Number Theory
• Counting and Probability
• Geometry
• Intermediate Algebra
• Precalculus

Each subject area contains five difficulty levels. Level 5 problems are difficult: many require insight that would challenge strong undergraduate students. The original paper reported that the best baseline at the time achieved only 6.9 percent, making it one of the most challenging benchmarks ever introduced for language models.

MATH is substantially harder than GSM8K. Early baseline performance was under 10 percent. State-of-the-art performance as of 2024 reaches 80 to 90 percent for frontier models, but level 5 problems remain challenging. A critical evaluation practice is to disaggregate performance by difficulty level.

The subject area breakdown is equally informative. Number theory and geometry tend to be harder for LLMs than pre-algebra and algebra. Number theory problems often require non-obvious divisibility arguments or modular arithmetic manipulations. Geometry problems sometimes require spatial reasoning that has no direct textual analog. These performance gaps guide training data collection: if a model is weak on geometry, adding more geometry problems and solutions to the training set directly targets the weakness.

The American Mathematics Competition (AMC) and American Invitational Mathematics Examination (AIME) are prestigious competitions. AMC problems are multiple-choice and span 30 questions in 75 minutes, requiring fast, creative problem solving. AIME problems are integer-answer problems with answers in the range 0 to 999, requiring multiple creative insights per problem, and the competition selects top AMC performers.

These benchmarks are valuable because they are difficult for current LLMs. AIME 2024 results placed frontier models at roughly human-level for average participants but well below top competitors. They also provide an important test for the memorization hypothesis: newly released competition problems cannot have been in training data. Consistently checking performance on the most recent AIME is more reliable than checking on older editions that models may have seen.

The AIME format creates an interesting evaluation design. With answers in range 0 to 999, random guessing has only 0.1 percent success rate, eliminating the possibility that models score well by chance. Every point on AIME reflects successful problem-solving. This is why AIME performance is often preferred over MATH percentage scores when trying to assess demonstrated capability rather than pattern matching.

Beyond competition math, researchers have constructed benchmarks for specific mathematical domains:

• Calculus: symbolic differentiation, integration, differential equations
• Linear algebra: matrix operations, eigenvalues, vector spaces
• Statistics: probability distributions, hypothesis testing, Bayesian inference
• Number theory: primality, modular arithmetic, divisibility

These domain-specific evaluations reveal that mathematical capability is not uniform. Models trained heavily on word problems may perform well on GSM8K but poorly on symbolic calculus. Benchmark selection should match the intended use case. A model being evaluated for use in a scientific computing context should be tested on calculus and linear algebra benchmarks, not just word problems.

There is also a growing recognition that standard benchmarks underrepresent certain areas of applied mathematics that matter practically. Financial mathematics, statistics, and discrete optimization appear less frequently in competition datasets than pure algebra and number theory. Practitioners building domain-specific applications often find it necessary to construct their own evaluation datasets drawn from domain-relevant problem types.