Chain-of-Thought › CoT Fine-Tuning

Few-shot and zero-shot CoT keep the model weights fixed and change only the prompt. An alternative is to train on reasoning chains directly, teaching the model to produce them as part of its standard output. CoT fine-tuning includes reasoning chains in the training data and optimizes the model to generate them.

Fine-tuning on CoT data was explored in the context of instruction tuning. Models fine-tuned on instruction datasets that include reasoning chains, such as chain-of-thought formatted examples from GSM8K, MATH, and other reasoning benchmarks, learn to spontaneously produce reasoning when asked questions that benefit from it.

The Flan series of models (Wei et al., 2022b; Chung et al., 2022) demonstrated that including chain-of-thought examples in instruction fine-tuning data improves reasoning performance even when reasoning is not explicitly requested at inference time. The model learns that careful, step-by-step reasoning is part of good response quality and applies it proactively. This is a meaningful shift: instead of requiring a trigger phrase or exemplars at inference time, the model develops a default behavior of reasoning thoroughly.

The benefit of CoT fine-tuning extends beyond the specific tasks included in training. Models trained on diverse reasoning chains from mathematics, science, and commonsense domains show improved reasoning on tasks that were not in the training mixture. The model appears to learn general reasoning habits, not just task-specific patterns. This transfer is more reliable when the fine-tuning data spans diverse reasoning types, and degrades when the training distribution is narrow.

An important application of CoT fine-tuning is knowledge distillation. Large models generate high-quality reasoning chains for a training corpus, and a smaller model is trained to reproduce both the chains and the conclusions. This transfers some of the reasoning capability of the large model to the smaller one.

Ho et al. (2022) showed that fine-tuning smaller models on CoT rationales generated by a larger model substantially improves the smaller model's reasoning accuracy, often matching or exceeding what few-shot CoT with the larger model achieves. The fine-tuned small model also produces reasoning chains at inference time, which means its outputs remain interpretable.

The quality of the generated reasoning chains matters a great deal for distillation. Chains from a teacher model that sometimes err propagate those errors into the student. Filtering generated chains by correctness before fine-tuning, or using only chains that arrive at the correct final answer, consistently improves distilled model quality. A practical procedure: use the teacher model to generate 8-16 candidate chains per training example, keep only those that arrive at the verified correct answer, and use those verified chains as fine-tuning data. This filters noisy signal and leaves a cleaner, more reliable training corpus.

The theoretical picture behind distillation is interesting. Small models cannot reliably do multi-step reasoning via few-shot CoT because, as noted earlier, the capability is emergent and only manifests at large scale in the prompted setting. But fine-tuning changes the weights as well as the context. A small model trained specifically to produce reasoning chains can internalize the decomposition pattern as part of its weights, partially circumventing the scale threshold. The fine-tuned small model is not equivalent to a large model with CoT, but it substantially outperforms the fine-tuned small model without CoT, and often outperforms the large model without CoT.

A more principled approach to training on reasoning chains is to reward correct process, not just correct outcomes. Process reward models (PRMs), explored by Lightman et al. (2023), provide supervision at the step level. Human annotators label each step in a reasoning chain as correct or incorrect, and a reward model trained on these labels scores steps rather than only final answers.

This step-level supervision addresses a core problem with outcome-only training: a chain can arrive at the correct answer through flawed reasoning (getting lucky on the final step after errors earlier), or arrive at an incorrect answer despite mostly correct reasoning (a small error late in the chain). Outcome reward models reinforce both of these undesirable patterns. PRMs reward good reasoning regardless of whether the final answer happens to be correct, and penalize bad reasoning even when the answer is right.

The practical impact is most visible when PRMs are combined with search. Rather than sampling a single chain, the system uses the PRM to guide a search process such as beam search, Monte Carlo tree search, or best-of-N sampling that selects chains with high step-by-step quality. Each candidate step is scored by the PRM before the search continues, allowing the search to prefer branches that are both logically sound and factually accurate.

This produced state-of-the-art results on competitive math benchmarks as of 2024. OpenAI's work on process supervision (Lightman et al., 2023) showed that PRM-guided search on the MATH benchmark substantially outperformed both outcome-reward models and unguided self-consistency sampling. The improvement was especially pronounced on the hardest problems, where a single error in any step is fatal and the solution space requires broad exploration rather than reliable execution of a known procedure.