Inverse Scaling › The Inverse Scaling Phenomenon

Standard scaling behavior follows a predictable pattern: as model size increases, loss decreases according to a power law. For most benchmarks, this translates to improved accuracy. The relationship appears so reliable that researchers routinely predict larger model performance from smaller models, and those predictions tend to be accurate. The scaling laws introduced in earlier chapters give us confidence that investing in scale will pay off.

Inverse scaling breaks this pattern. On certain tasks, the performance curve inverts. Rather than accuracy improving with scale, it degrades. The models are not just failing to improve; they are actively getting worse in a way that becomes more pronounced as the model grows larger. What distinguishes inverse scaling from ordinary variance is its consistency: the same degradation appears across GPT-3 model sizes from 125 million to 175 billion parameters, across different model families from different organizations, and across multiple independent evaluations. This cross-model consistency is the scientific signature that something real and systematic is at work.

Think of inverse scaling like a student who has memorized so many example solutions that they can no longer think flexibly. A first-year student with limited memorized patterns must reason from first principles when encountering a novel problem. A third-year student who has memorized hundreds of problem templates may instead pattern-match to the most similar template they know, even if that template does not quite fit. Their greater knowledge works against them when the problem requires deviating from familiar patterns. The third-year student remains more capable overall: on most problems, their deep familiarity with templates gives them a decisive advantage. The failure emerges specifically on problems that are engineered to look like familiar templates but require a different approach.

Before presenting the formula, let us consider what we are trying to capture. Normally, we expect a larger model to perform better, so accuracy would be a positive function of model size. But on inverse scaling tasks, we observe the opposite: accuracy decreases as models grow. We need a mathematical expression that formalizes this counterintuitive behavior, showing how performance deteriorates systematically rather than randomly as scale increases. The formula should reveal whether degradation is present, its speed, and its functional form, giving us a tool for comparing the severity of inverse scaling across different tasks and task categories.

The following formula captures how accuracy degrades as model size grows:

$$\text{Accuracy}(N) \propto N^{-\alpha}$$

where:

• $\text{Accuracy}(N)$: the model's performance on the task as a function of model size
• $N$: the number of parameters in the model
• $\alpha$: a positive constant ($\alpha > 0$) representing the rate of performance degradation
• $\propto$: indicates proportionality (accuracy scales with the inverse power of $N$)

The negative exponent is the key insight that makes this formula so revealing. Consider what happens as $N$ grows larger. When we raise an increasing number to a negative power, the result shrinks. Mathematically, $N^{-\alpha}$ is equivalent to $\frac{1}{N^{\alpha}}$, so as $N$ increases, the denominator grows and the overall value decreases. This is the mathematical signature of inverse scaling. Larger models do not just fail to improve; they actively get worse in a predictable, quantifiable way.

Why does this formula make sense intuitively? Notice that the proportionality relationship tells us the degradation is not erratic. It follows a clean power-law structure, which means we can predict how much worse a larger model will be given the smaller model's performance and the value of $\alpha$. This predictability is itself scientifically important: it rules out explanations based on random noise or implementation differences and points instead to a systematic mechanism built into how models learn from data. If the degradation were random, we would expect different model families to show different directions of change. The fact that they all show the same direction, with performance consistently declining as size increases, demands a structural explanation.

The parameter $\alpha$ controls the severity of the inverse scaling effect. A larger value of $\alpha$ means that accuracy drops more steeply as model size increases. For example, if $\alpha = 0.1$, the degradation is gradual, while $\alpha = 0.5$ would represent a much more dramatic decline. Different tasks exhibit different values of $\alpha$. This reflects how strongly each task's requirements conflict with the patterns that scale amplifies. Tasks where the conflict is mild show small $\alpha$; tasks where the conflict is basic to how models learn from language show large $\alpha$. This variation in $\alpha$ across tasks is itself informative: it tells us which aspects of model behavior are most deeply encoded and therefore most resistant to correction at inference time.