GSM8K › Accuracy Metrics for GSM8K

Evaluating solutions to GSM8K problems requires precise answer extraction and comparison. Unlike open-ended generation tasks where BLEU or ROUGE scores might apply, mathematical problems demand exact correctness. There is no partial credit for being close to the right answer; the binary nature of mathematical truth requires binary evaluation criteria.

The evaluation protocol must handle the tension between the richness of natural language generation and the precision required for mathematical verification. Models may produce verbose explanations, include units, or format numbers differently, yet the underlying mathematical claim must be extracted and verified against the ground truth with perfect accuracy.

The primary metric for GSM8K is exact match on the final numeric answer. An answer is considered correct only if it matches the ground truth exactly, with no tolerance for rounding errors or alternative representations. If the correct answer is 18, then "18 dollars," "18.0," or "eighteen" would typically be marked incorrect unless the evaluation script specifically normalizes these variations.

Let $N$ denote the number of test problems, let $y_i$ be the ground truth answer for problem $i$, and let $\hat{y}_i$ be the model's predicted answer after extraction and normalization. The GSM8K accuracy metric is:

$$\text{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = y_i]$$

where $\mathbf{1}[\hat{y}_i = y_i]$ equals 1 when the predicted answer exactly matches the ground truth and 0 otherwise.

This strict criterion shows the binary nature of mathematical truth. A reasoning chain that arrives at "17" instead of "18" for the duck egg problem contains an error somewhere, whether in arithmetic or problem understanding, and fails to solve the problem correctly. While this strictness may seem harsh compared to human grading where partial credit is common, it gives an unambiguous signal for model capability assessment and avoids the subjective judgment calls required for partial credit assignment.

Robust evaluation requires parsing model outputs to extract numeric answers. Models may generate verbose responses including explanations, units, and commentary. The extraction system must handle this complexity to isolate the mathematical claim from its linguistic packaging.

Standard extraction strategies include:

• Regular expression matching for numbers following the #### delimiter (when models are prompted to use this format)
• Extracting the final number appearing in the text, based on the heuristic that models typically state their final answer last
• Pattern matching for phrases like "The answer is [number]" or "Therefore, the result is [number]"

The extraction process materially impacts reported accuracy. If a model correctly calculates an answer but expresses it as "18 dollars" while the evaluator expects "18," the model receives no credit despite possessing the correct understanding. This sensitivity necessitates careful normalization routines that strip units, handle decimal representations, and account for common formatting variations while remaining reliable to adversarial cases where numbers in the explanation might be mistaken for the final answer.

Normalization typically applies a standardized pipeline: remove currency symbols, strip units, replace commas in large numbers (e.g., "1,024" becomes "1024"), and convert decimal representations to standard form. The challenge is that overly aggressive normalization can introduce false positives. If a problem asks for the answer in cents and the model answers in dollars, stripping units might make an incorrect answer appear correct.

Beyond final answer correctness, researchers sometimes analyze per-step accuracy by comparing generated reasoning chains against reference solutions. This finer-grained metric identifies where reasoning breaks down and gives diagnostic insight into model limitations.

Four distinct error types characterize reasoning failures on GSM8K:

Conceptual errors stand for misunderstandings of the problem structure itself: adding instead of subtracting, using a multiplication rate where a division is required, or misidentifying which quantities are needed to the solution. These errors occur regardless of arithmetic accuracy; the model understands all the numbers but connects them with the wrong operation.

Arithmetic errors occur when the model correctly identifies the required operations but computes them incorrectly. The model understands that it needs to multiply 9 eggs by $2 per egg, but arrives at $17 instead of $18 due to a calculation mistake. These errors are purely computational rather than conceptual.

Truncation errors involve stopping before reaching the final answer, perhaps after calculating an intermediate value. The model correctly computes that Janet has 9 eggs remaining, but fails to multiply by the price to obtain the final dollar amount.

Hallucination errors involve introducing quantities not present in the problem or inventing calculations without basis in the problem text. A model might fabricate a "processing fee" or introduce a fictional discount that changes the problem structure entirely.

Analyzing error patterns reveals that larger models primarily reduce arithmetic errors while maintaining similar rates of conceptual misunderstanding, suggesting that scale improves execution more than comprehension. This decomposition is important for guiding model improvement: if errors are primarily arithmetic, integrating calculators or improving numerical precision may help; if errors are conceptual, better training on problem understanding or logical reasoning may be required.