Emergence vs Metrics › Re-examining Emergence Claims

In 2023, researchers at Stanford and Google published a provocative paper titled "Are Emergent Abilities of Large Language Models a Mirage?" They systematically tested whether apparent emergence could be explained by metric choice alone. The paper made an unusually direct argument: the researchers showed that a researcher can create the appearance of emergence in any dataset simply by choosing an appropriate metric, and can make emergence disappear by choosing a different metric. If emergence were a real phenomenon, switching metrics should not cause it to vanish entirely.

The researchers took tasks previously claimed to show emergence and re-evaluated models using continuous metrics. They found that for nearly all tasks examined:

1. Accuracy showed sharp transitions consistent with prior emergence claims
2. Continuous metrics showed smooth, predictable improvement across all scales
3. The transition point was predictable from the metric and task structure

This pattern held across arithmetic, translation, word unscrambling, and other tasks that had been cited as evidence for emergent capabilities. The researchers also showed the complementary result: if you take a dataset where capabilities appear to develop smoothly, you can manufacture apparent emergence by switching to a harder metric that requires near-perfect token-level accuracy. The "emergence" appears or disappears based on measurement choices, not based on anything the model does differently.

The experimental design was careful to use the same model weights and the same tasks throughout. Nothing about the model changed between the "emergence" and "no emergence" conditions. Only the metric changed. This is the strongest possible evidence for the measurement artifact hypothesis.

The mirage paper made an important distinction between two types of emergence:

Metric-induced emergence occurs when smooth underlying improvements appear discontinuous due to threshold effects in measurement. This is not truly "emergent" in any meaningful sense; the capability develops gradually and predictably. The transition point can be calculated in advance from the metric structure and the expected improvement trajectory.

True emergence would involve discontinuous changes in underlying capability: qualitative changes in how the model processes information that couldn't be predicted from smaller-scale behavior. True emergence would persist regardless of which metric you use to measure it, because the discontinuity would be in the underlying model behavior rather than in the measurement lens.

The paper argued that most documented cases of emergence are metric-induced, not true emergence. This does not mean the capabilities are unimportant or that scaling does not produce remarkable abilities. It means the discontinuity is in our measurement, not in the model's developmental trajectory. The capabilities were developing all along; we simply couldn't see them with the tools we were using.