Mathematical Reasoning › Mathematical Reasoning Training

Beyond prompting, substantial research has gone into fine-tuning models specifically for mathematical reasoning. Several distinct approaches have emerged, each with different assumptions about what is available and what the target capability is.

The most direct approach is fine-tuning on large collections of mathematical problems with solutions. The key variables are data quality, data diversity, and solution format.

Solution format matters significantly. Models trained on solutions that show only final answers learn something different from models trained on step-by-step derivations. The latter acquire explicit reasoning patterns that generalize better to novel problems. Research from DeepMind, Meta, and others consistently finds that training on chain-of-thought solutions substantially outperforms training on answer-only data.

There is an important subtlety about data quality versus quantity. Early mathematical fine-tuning research found that a small dataset of high-quality step-by-step solutions outperformed a large dataset of answer-only examples. This is analogous to the difference between a textbook that shows working and a textbook that only provides final answers. The model learns to emulate the reasoning format it sees during training. If that format is rich and explicit, the model develops richer reasoning patterns. If it is sparse, the model learns to produce short, answer-focused generations that fail on harder problems.

Data diversity means exposure to many problem types and difficulty levels. A model trained only on algebra problems will not learn proof techniques. Mixing problems from GSM8K, MATH, competition archives, and textbooks produces mathematical reasoning that transfers across more problem types because the model encounters the full range of problem structures and solution strategies. Practitioners often find that including problems at the edge of the model's current capability produces the best improvement: problems that are too easy provide no signal, and problems that are too hard provide no correct solutions to learn from.

Synthetic data generation has become an important tool for scaling mathematical training corpora. Starting from a base set of problems, researchers use capable models to generate step-by-step solutions, then filter for correctness using a CAS or known answers. This allows training corpora to grow far beyond what human annotation could provide, at the cost of introducing systematic errors in the generated reasoning that must be managed carefully. The filtering step is critical: training on incorrect solutions with plausible-looking reasoning is worse than training on no data at all, because the model learns confident but wrong reasoning patterns.

The Metamath dataset and similar resources take a different approach: they mine formal mathematical libraries for true statements and use automated proof-checking to verify correctness. This eliminates the filtering problem but produces a different style of reasoning, formal and verbose rather than pedagogical, which may not transfer as well to informal problem-solving contexts.

RL provides a natural framework for mathematical reasoning because mathematical problems have clear success criteria. A model can attempt many solutions, receive binary correctness signals, and learn to increase the probability of correct approaches.

The DeepSeekMath model family (2024) applied group relative policy optimization (GRPO) to mathematical reasoning, using only correctness of final numerical answers as the reward signal. The results were striking: RL fine-tuning from a supervised baseline substantially improved performance on MATH, even though the reward signal was sparse. The RL model learned to generate longer, more careful reasoning chains, analogous to a student who learns from getting problems wrong that they need to show their work more explicitly.

Mathematically, GRPO operates on groups of sampled outputs. For each problem, the model generates $G$ candidate solutions. The relative advantage of each solution is computed against the group average, normalizing for problem difficulty. The policy gradient update encourages solutions that scored above average and discourages those that scored below.

$$\mathcal{L}_{\text{GRPO}}(\theta) = -\mathbb{E}\left[\frac{1}{G} \sum_{i=1}^{G} A_i \log \pi_\theta(o_i \mid q)\right]$$

where:

• $\theta$: the model parameters being updated
• $G$: the number of sampled outputs per problem
• $A_i$: the advantage of output $i$ relative to the group mean reward
• $\pi_\theta(o_i \mid q)$: the probability of generating output $i$ given question $q$ under the current policy

The advantage $A_i$ is computed as:

$$A_i = \frac{r_i - \text{mean}(\{r_1, ..., r_G\})}{\text{std}(\{r_1, ..., r_G\})}$$

where $r_i$ is the binary reward (1 if correct, 0 if not) for the $i$-th generated solution.

This normalization has an important effect: it makes the signal invariant to the absolute difficulty of problems. A problem where the model gets 8 out of 10 attempts correct and a problem where it gets 2 out of 10 produce comparable gradient magnitudes, focusing learning on the relative quality of solutions rather than raw success rates. This matters for mixed-difficulty training sets: without normalization, easy problems with consistently high rewards dominate the gradient and the model learns to solve them better at the expense of harder problems where improvement is most needed.

Standard policy gradient methods like REINFORCE have high variance because rewards are not normalized. A single problem that the model happens to solve correctly on one run but not others creates noisy, high-magnitude gradient updates. GRPO stabilizes this by computing advantages within a group of rollouts for the same problem, treating the group mean as a baseline. The resulting gradients have much lower variance, allowing higher learning rates and faster convergence.

The group size $G$ is a critical hyperparameter. Small groups (G=4) have high advantage variance because the mean is estimated from few samples. Large groups (G=64) give more stable advantage estimates but require more inference compute. Most practitioners find G=8 to G=16 to be a practical sweet spot that balances computational cost with training stability.

An emerging line of work uses iterative self-play to improve mathematical reasoning without additional human annotation. The model generates solutions to problems, checks them against known correct answers, and uses its own errors as training signal.

The ReST (Reinforced Self-Training) approach generates many solution candidates per problem, filters for correct solutions, and fine-tunes on these. This bootstraps from an initial supervised model to progressively better performance. The key insight is that as the model improves through each iteration of self-training, it can solve problems that were previously out of reach, which expands the set of training examples available for subsequent iterations.

The critical challenge is the coverage problem: if the initial model cannot solve a problem at all, no correct solutions exist to learn from, and the problem remains a blind spot throughout training. One mitigation is to include hint injection, where the model is given partial solutions or key steps and must complete the reasoning. This increases coverage on harder problems at the cost of requiring the hint-generation procedure.

A complementary approach to improving mathematical capability is tool augmentation: giving models access to a Python interpreter or CAS that can handle precise arithmetic and symbolic computation. Instead of computing 3,478 plus 19,204 in text (which LLMs do poorly), the model generates code that calls a calculator. Instead of integrating $\int x e^{x^2} dx$ symbolically in text, the model writes a SymPy call.

Tool augmentation sidesteps the arithmetic brittleness and symbolic manipulation weaknesses of LLMs without requiring those weaknesses to be fixed through training. The model's role shifts from doing the computation to orchestrating it: understanding the problem structure, selecting the right approach, generating the right code, and interpreting the results.

Models like GPT-4 Code Interpreter (now Advanced Data Analysis) demonstrate this paradigm clearly. When given a complex mathematical problem, the model writes Python code to solve it, executes that code, receives the output, and incorporates the result into its reasoning. Performance on problems that benefit from computation jumps dramatically compared to text-only reasoning, because exact computation is guaranteed correct by the interpreter.

The limitation of tool augmentation is that it does not help with problems where the insight required cannot be expressed as code. "Prove that there are infinitely many primes" cannot be reduced to a computation. The model still needs conceptual understanding for such problems. Tool augmentation is most effective for problems with a computational core surrounded by linguistic interpretation.