Emergence in Neural Networks › The Emergence Debate

Not all researchers accept that emergence is a real phenomenon. A critique, articulated most forcefully by Schaeffer et al. (2023), argues that emergence might be an artifact of how we measure capabilities.

The central argument is that emergent behavior appears when we measure performance with metrics that are:

1. Nonlinear in the underlying model competence
2. Discontinuous at certain thresholds (like exact-match accuracy)

Consider exact-match accuracy for a multi-step task. Even if a model's probability of getting each step correct improves gradually, the probability of getting all steps correct (which is what exact-match measures) follows a nonlinear transformation. If we denote per-step accuracy as $p$ and require $n$ steps for exact match, then:

$$\text{Exact\_Match\_Accuracy} = p^n$$

where:

• $p$: the probability of getting any single step correct (assumed equal across steps for simplicity)
• $n$: the number of steps required for a complete correct answer
• $p^n$: the probability that all $n$ independent steps are correct

This exponential relationship is the mathematical heart of the metric critique. When $p$ is below 1, raising it to a power $n$ compresses the scale sharply. A per-step accuracy of 0.8 yields exact-match of $0.8^5 = 0.33$ for a 5-step task, but $0.8^{10} = 0.11$ for a 10-step task. The same underlying competence looks very different depending on task length.

To build intuition for why this matters, imagine watching someone practice juggling. Their ability to keep one ball in the air improves steadily from 50% success to 90% over months of practice. But if you only measure "can you juggle three balls for 30 seconds," you'd see near-zero success for a long time, then a sudden jump to competence. The underlying skill improved gradually; the all-or-nothing measurement created the appearance of sudden emergence.

The same principle applies to language model evaluation. If a model needs to get a five-step reasoning chain entirely correct to receive credit, and its per-step accuracy is improving from 0.7 to 0.95, the exact-match score transforms from $0.7^5 \approx 0.17$ to $0.95^5 \approx 0.77$. That looks like emergence, from barely above random to quite good. But the per-step improvement was smooth throughout.

Let's illustrate this with a concrete example:

The visualization reveals how metric choice shapes our perception. The underlying competence (per-step accuracy) improves smoothly, but exact-match accuracy shows an apparent phase transition. The more steps required, the sharper the apparent emergence. This isn't the model suddenly acquiring a new capability; it's the same gradual improvement viewed through a lens that exaggerates the transition.

Schaeffer et al. demonstrated that for many "emergent" tasks, using continuous metrics (such as token-level accuracy or Brier scores) instead of exact-match accuracy reveals smooth, predictable scaling. The emergence "disappears" when measured differently.

This finding has measurable implications. If emergence is a metric artifact, then:

• We can predict capabilities at larger scales from smaller model performance
• There is no basic unpredictability in capability development
• The concern about "surprising" new capabilities may be overstated

However, the counterargument is that for practical purposes, it matters whether a model can complete a task correctly, not whether it's making gradual progress on subtasks. A model that gets 50% of arithmetic steps correct still produces wrong answers. For the user, the task-level capability does emerge.

The debate may be partially resolved by distinguishing between:

• Metric-induced emergence. The appearance of sudden transitions due to nonlinear metrics. This is a measurement artifact.

• True capability transitions. Qualitative changes in what a model can do, such as the ability to perform in-context learning or use chain-of-thought reasoning. These might involve actual phase transitions in the network's computations.

Some capabilities may show both types. For example, chain-of-thought prompting changes how a model processes information, a qualitative transition, but the measurable improvement might also be amplified by the exact-match metrics used to evaluate it.

The emergence debate has practical implications. If emergent capabilities are unpredictable, then larger models might develop surprising and potentially dangerous capabilities. This motivates caution in scaling and extensive evaluation at each new scale.

If emergence is largely a metric artifact, the situation is more manageable. Careful evaluation with continuous metrics could reveal gradual capability development, letting better prediction and control.

The truth likely lies between these extremes. Some capabilities do appear relatively suddenly (if not truly discontinuously), while others improve gradually. Understanding which is which requires careful empirical study.