GSM8K › Dataset Structure and Problem Characteristics

GSM8K consists of 8,500 high-quality grade school math word problems, split into 7,500 training examples and 1,000 test examples. Each problem requires between two and eight steps to solve, involving basic arithmetic operations (addition, subtraction, multiplication, and division) with natural number solutions. The problems are linguistically diverse, presenting scenarios ranging from retail transactions and travel planning to work rates and age relationships.

The dataset designers deliberately constrained the mathematical complexity to elementary operations while maximizing the reasoning depth required to reach solutions. This design choice isolates reasoning capability from advanced mathematical knowledge. This keeps failures indicate limitations in logical processing rather than gaps in mathematical education. By focusing on natural numbers and basic operations, GSM8K tests whether models can combine simple tools in complex ways, much like observing whether a carpenter can build detailed furniture using only basic hand tools.

The dataset captures several categories of mathematical reasoning that reflect the cognitive demands of early mathematics education. These categories map to distinct mental models that humans develop when learning to translate real-world situations into mathematical formalism.

Multi-step arithmetic problems require sequential operations where the output of one calculation is input to the next. Consider: "John buys 3 packs of gum for $2 each and gives the cashier a $20 bill. How much change does he receive?" This requires calculating the total cost (3 \times \2 = $6$) before determining the change ($$20 - $6 = $14$). The cognitive load here involves maintaining intermediate results across steps. This keeps the calculation of total expenditure completes before attempting to compute the difference from the initial amount. When models collapse these steps into a single inference step, they lose the verification opportunity that explicit intermediate computation gives.

Rate and proportion problems involve speeds, work rates, or ratios. Consider: "If 5 workers can build a wall in 10 days, how long would 10 workers take?" The solution requires recognizing the inverse relationship between workers and time, understanding that doubling the workforce halves the duration. This tests whether models grasp proportional reasoning or merely memorize formulas. The linguistic packaging of rate problems varies enormously, with speed problems, efficiency problems, and concentration problems all involving the same underlying proportional structure expressed in very different surface forms.

Part-whole relationship problems require decomposition of quantities into components or aggregation of parts into totals. These test whether models can track how subsets relate to larger groups without double-counting or omitting elements. Calculating total expenses across multiple categories, or determining remaining inventory after several allocations, requires maintaining a mental inventory of which quantities have been accounted for. These problems are particularly prone to one specific error: reusing a quantity that was already consumed or redistributed in an earlier step.

Temporal sequencing problems involve time calculations across different periods, requiring models to maintain chronological order and properly aggregate durations. These problems test the ability to handle operations where order matters, such as calculating end times given start times and durations, or determining meeting times given different travel schedules. The non-commutativity of time arithmetic, combined with unit conversions (minutes to hours, days to weeks), creates multiple opportunities for reasoning failure.

Each GSM8K example includes both the problem text and a detailed natural language solution. The solutions are structured to show step-by-step reasoning, with the final answer explicitly indicated after a #### delimiter. This format proves important for both human readability and automated evaluation:

The #### delimiter precedes the final numeric answer, letting automated extraction while the preceding text gives the reasoning chain. This structure serves dual purposes: it allows automated evaluation systems to parse answers unambiguously while giving rich supervision signals for training models to generate coherent reasoning traces. The explicit separation of reasoning from final answer also facilitates error analysis, letting researchers to determine whether models fail due to comprehension errors, arithmetic mistakes, or answer formatting issues.

The annotations were written by human contractors trained specifically for this task. The quality control process involved multiple review rounds to ensure that each solution was numerically correct and pedagogically clear, meaning each intermediate step was explicitly stated with its arithmetic shown. This consistency in annotation style proved important for training: models learning from well-structured, consistently formatted reasoning traces develop stronger capabilities for generating their own structured reasoning.

The 2-to-8 step range in GSM8K creates a real difficulty spectrum within the bounds of elementary mathematics. Two-step problems resemble direct arithmetic exercises where the challenge is primarily parsing the linguistic description. Eight-step problems require maintaining and correctly updating several intermediate quantities while working through multiple layers of problem structure.

This distribution reveals something important about how reasoning errors compound. If a model solves each individual step with 90% accuracy, an independent error model predicts the probability of solving an $n$-step problem correctly as $0.90^n$. For a 2-step problem, this gives 81% accuracy. For an 8-step problem, it gives only 43% accuracy. The actual decay is even steeper in practice because errors in early steps corrupt the input to later steps, causing downstream failures that cascade through the reasoning chain. This compounding nature of reasoning errors makes the 8-step problems disproportionately challenging relative to their individual step complexity.