Scaling Frontiers › Scaling and Reasoning
Language AI HandbookAll chapters Scaling Frontiers › Scaling and ReasoningOne section · Sections in this chapter6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Perhaps the most consequential recent development in the scaling conversation is the growing evidence that reasoning capability scales differently from knowledge retrieval. This distinction matters because it suggests that future progress may require scaling existing architectures and rethinking what kind of computation models perform.
The Reasoning Gap
Language models are remarkably effective at retrieving and recombining information from training data. Factual questions, translation, summarization, and stylistic tasks have all seen impressive gains from scaling. But for tasks that require multi-step reasoning, where the correct answer depends on a chain of logical deductions rather than pattern matching, the picture has been more mixed.
Mathematical problem-solving illustrates this clearly. Models can recite mathematical facts and solve standard textbook problems with high accuracy. But novel mathematical competition problems, which require applying reasoning in combinations not likely seen verbatim in training data, have historically been much harder. The model has seen countless examples of mathematical reasoning, so the gap lies in composing those patterns into novel chains.
The gap has narrowed substantially with reasoning-augmented models. Chain-of-thought prompting, reinforcement learning on verifiable tasks, and extended thinking all improve multi-step reasoning performance. But the mechanisms underlying this improvement are still debated. Is the model learning reusable reasoning procedures, or is it learning to more systematically retrieve and combine patterns from training data?
Verifiable Tasks and RLHF
One productive approach to improving reasoning capability is to train models on tasks where correctness can be verified programmatically. Mathematical problems have exact answers. Code either passes tests or it does not. This verifiability creates a clean learning signal that is difficult to fake: the model cannot simply pattern-match to what "looks like" a correct answer; the answer must be correct.
Training with reinforcement learning on verifiable tasks has shown strong results. DeepSeek-R1, trained primarily on mathematical and code generation tasks with verified rewards, achieved competitive performance with much larger models on reasoning benchmarks. The key insight is that the quality of the reward signal may matter as much as its quantity: a sparse but perfectly accurate reward from a verifier is more informative than a dense but noisy reward from a human preference model.
This opens a potentially important pathway beyond the data wall. If models can generate candidate solutions to verifiable problems and receive accurate reward signals from automated checkers, then the amount of usable training signal scales with the compute used for generation and verification, not with the supply of human-labeled data. This is sometimes called "self-play" for language models: the model learns from its own attempts, guided by an objective judge.
import numpy as np
# Simulate the learning curve under different reward signal quality conditions
# Higher quality reward signal = faster improvement per training step
def learning_curve(steps, signal_quality, noise_level):
"""
Simplified model of performance improvement as a function of:
- steps: number of training steps
- signal_quality: how accurate the reward signal is (0 to 1)
- noise_level: variability in the reward signal
"""
np.random.seed(42)
noise = np.random.normal(0, noise_level, len(steps))
performance = 1.0 - np.exp(-signal_quality * steps / 1000) + noise
return np.clip(performance, 0, 1)
steps = np.linspace(0, 5000, 200)
# Perfect verifier (e.g., code test suite, math checker)
perfect_verifier = learning_curve(steps, signal_quality=2.0, noise_level=0.01)
# Human preference model (noisy but dense)
human_preference = learning_curve(steps, signal_quality=1.0, noise_level=0.03)
# Weak heuristic (e.g., length-based reward)
weak_heuristic = learning_curve(steps, signal_quality=0.3, noise_level=0.05)
Scalable Oversight and the Alignment Challenge
As models become more capable, evaluating their outputs becomes harder. A model that outperforms its evaluators on a complex reasoning task cannot be reliably judged by those evaluators. This creates a feedback loop problem: the models that most need rigorous oversight are the ones where rigorous oversight is hardest to apply.
Scalable oversight is an active research area aimed at solving this problem. The core idea is to use AI assistance to help human evaluators assess AI outputs that would otherwise be beyond human comprehension. Debate between AI systems, where models argue opposing positions and humans judge the quality of the arguments rather than the conclusion, is one proposed approach. Recursive reward modeling, where models learn to decompose complex tasks into subproblems that humans can evaluate more easily, is another.
These approaches are important for scaling because they determine whether the training signal for future, more capable models can remain aligned with human values and intentions. If the reward model used to guide training via RLHF becomes outdated relative to the model being trained, the model may optimize for rewards in ways that diverge from actual human preferences. This is sometimes called reward hacking or Goodhart's Law in this context: when a measure becomes a target, it ceases to be a good measure.