Inverse Scaling › Inverse Scaling: When Larger Language Models Perform Worse

The scaling laws we explored in Part XXIII: Scaling Laws show that more parameters, more compute, and more data consistently improve model performance across a wide range of benchmarks. The Kaplan and Chinchilla scaling laws describe predictable power-law relationships between scale and loss. This narrative, that bigger is better, has driven the development of increasingly massive language models. Researchers and engineers have internalized scaling as an axiom: when in doubt, train a larger model on more data.

But what if scale sometimes makes models worse?

Inverse scaling is a phenomenon where larger language models perform worse than smaller ones on specific tasks. Performance actively degrades as models grow. A 175 billion parameter model might get a task wrong that a 1 billion parameter model handles correctly. This is not a matter of noise or statistical fluctuation. Researchers have documented reliable, replicable cases where the degradation follows a systematic pattern across multiple model families and architectures. The pattern is reproducible and exposes how training on text corpora shapes model behavior: larger models fail in predictable, structured ways rather than at random.

This chapter examines why inverse scaling occurs, what tasks trigger it, and what these findings reveal about how language models learn. Understanding inverse scaling affects AI safety, deployment decisions, and models of what neural networks optimize. If you have ever deployed a larger model expecting improvements and been surprised by unexpected failures, inverse scaling may explain why. More broadly, it forces a reconsideration of what it means for a model to be "better" and what evaluation suites must cover to give a trustworthy picture of model quality.

The core insight is this: larger models do not learn "better" in some generic sense. They learn the patterns in their training data more thoroughly and apply them more consistently. Usually that helps. But certain carefully constructed tasks expose situations where the patterns that training amplifies are precisely the wrong ones to apply. Scale makes models more capable and more committed, and that commitment becomes a liability on tasks designed to require flexibility and override of learned defaults. The analogy is illuminating: think of a highly experienced lawyer who has internalized so many precedents that they struggle to evaluate a novel case on its own terms, because every fact pattern retrieves a familiar precedent and they reach for it automatically.

We will work through the mechanics of inverse scaling systematically: first building mathematical intuition for the phenomenon, then examining the categories of tasks that trigger it, exploring the theoretical frameworks researchers have proposed to explain it, and finally developing practical strategies to detect and mitigate it in real deployments. Along the way, we will see how inverse scaling connects to the emergence phenomena covered in the previous chapter and how it foreshadows the grokking dynamics discussed in the next one.