Emergence in Neural Networks › Mechanisms of Emergence

Why do certain capabilities appear suddenly rather than gradually? Several hypotheses attempt to explain emergence, each giving a different perspective on how neural networks acquire complex abilities.

Many tasks require composing multiple sub-capabilities. Consider three-digit addition:

1. Parse the digits from text
2. Add corresponding digit pairs
3. Handle carry operations
4. Format the result as text

If each sub-capability has its own scaling threshold, the full task only succeeds when all sub-capabilities are above threshold. This creates a "weakest link" dynamic where the overall task appears to emerge suddenly, even if each component improves gradually.

To understand this mathematically, consider what happens when success requires multiple independent conditions to all be met. If each sub-capability succeeds with probability $p_i$, and the full task requires all $n$ sub-capabilities to function correctly, then:

$$P(\text{task success}) = \prod_{i=1}^{n} p_i$$

where:

• $P(\text{task success})$: probability of completing the full task correctly
• $p_i$: probability that sub-capability $i$ succeeds (a value between 0 and 1)
• $n$: number of required sub-capabilities
• $\prod_{i=1}^{n}$: the product operator, multiplying together all $p_i$ values from $i=1$ to $n$

The product form reveals an important mathematical property: even when each individual probability is reasonably high, their product can be surprisingly low. This happens because multiplying probabilities less than 1 always yields a smaller number. For instance, if four sub-capabilities each have 90% success, the task success rate is only $0.9^4 \approx 65\%$. With six sub-capabilities at 90% each, success drops to about 53%. The more components a task requires, the more stringent the demands on each component's reliability.

This compositional structure affects how emergence appears in practice. Imagine that each sub-capability follows its own sigmoid improvement curve as the model scales, but these curves have different inflection points. The first sub-capability might become reliable at 1 billion parameters, the second at 3 billion, the third at 8 billion, and the fourth at 20 billion. The overall task can only succeed when all four are functional. This means the task as a whole only becomes reliable once the last sub-capability crosses its threshold. An observer watching only the overall task accuracy would see nothing but near-random performance until approximately 20 billion parameters, then a sudden jump to success. The individual improvements were gradual, but their combination appears discontinuous.

Let's simulate this compositional effect:

Another hypothesis draws on mechanistic interpretability research. Neural networks may need to form specific computational circuits to solve certain tasks. Below a necessary scale, the network lacks the capacity to stand for these circuits. Above that scale, the circuits can form and the capability appears.

This view aligns with research identifying specific attention patterns and MLP computations responsible for particular capabilities. For example, researchers have identified "induction heads" in transformers: attention patterns that let in-context learning by copying patterns from earlier in the context.

The circuit formation hypothesis suggests that emergence is in effect a phase transition in the network's representational capacity: a sudden move from "cannot stand for the solution" to "can stand for the solution."

A third hypothesis focuses on the optimization surface. As models scale, their loss surfaces may undergo qualitative changes. At small scales, the global minima accessible during training might not include the weights needed for certain capabilities. At larger scales, new regions of weight space become accessible, containing solutions to previously unsolvable tasks.

This connects to the observation that larger models are often easier to optimize despite having more parameters. The increased dimensionality might smooth the loss surface, which makes it easier to find good solutions.

Some researchers argue that emergence relates to the information content of tasks. A task requiring $k$ bits of "algorithmic information" to solve can only emerge when the model has enough parameters to encode that information. This view predicts that more complex tasks (requiring more algorithmic information) will emerge at larger scales, which matches empirical observations.