GSM8K › Worked Example: Tracing a Multi-Step Reasoning Chain

To build concrete intuition for how chain-of-thought evaluation works in practice, let us trace through a complete GSM8K problem step by step.

Problem: "A baker has 120 cookies. She packs them into boxes of 8 and sells each box for $3.50. However, 5 cookies are broken and cannot be sold. How much money does the baker earn?"

Reading the problem, we identify the following needed quantities:

• Total cookies: 120
• Broken cookies: 5
• Cookies per box: 8
• Price per box: $3.50

We also identify the implicit structure: broken cookies must be removed before packing, and only complete boxes can be sold. The question asks for total revenue from box sales.

A well-formed chain-of-thought solution proceeds as follows:

The number of sellable cookies is $120 - 5 = 115$.

The number of complete boxes is $\lfloor 115 / 8 \rfloor = 14$ boxes (since 115 divided by 8 is 14 with remainder 3, and partial boxes cannot be sold).

The total revenue is 14 \times \3.50 = $49.00$.

This problem contains several traps that lead to common errors:

The floor division step is the most common failure point. A model that divides 115 by 8 and gets 14.375 might round to 14 or 15 depending on how it handles the non-integer result. Rounding up to 15 is a conceptual error: you cannot sell 15 boxes with only 115 cookies when each box requires 8. This specific trap tests whether the model understands the discrete nature of the packing constraint.

A secondary failure occurs if the model forgets to subtract the broken cookies before dividing. Dividing 120 by 8 gives 15 boxes, yielding a revenue of $52.50. This error shows a truncation of the problem structure, skipping the constraint that broken cookies cannot be sold.

A third failure mode involves arithmetic errors in the final multiplication. Computing $14 \times 3.50$ requires careful decimal arithmetic, and a model that treats this as $14 \times 3 + 14 \times 0.5$ must correctly add $42 and $7 to obtain $49. Each sub-computation is a potential error site.

The ground truth answer is 49 (representing $49.00). An evaluation script extracts the final number from the model's response. If the model correctly solves the problem and formats its answer as "The baker earns $49.00," the normalization pipeline strips the dollar sign and decimal to extract "49." The comparison $\hat{y} = y = 49$ succeeds, and the problem is marked correct.

If the model instead answers "The baker earns $52.50" because it forgot the broken cookie constraint, the extracted answer is "52.50" or "5250" depending on normalization, and the comparison fails. The error is marked as a conceptual error (incorrect problem structure) rather than an arithmetic error.