Emergence in Neural Networks › Examples of Emergent Capabilities

Researchers have documented numerous capabilities that appear to emerge with scale. Let's examine several prominent examples, grouped by category.

Basic arithmetic gives clean examples of emergence because success is unambiguous: the answer is either correct or incorrect.

Multi-digit addition. Models below a certain scale cannot reliably add three-digit numbers. They might get single-digit addition correct (perhaps through memorization from training data), but fail on larger numbers. Above the threshold, performance jumps sharply.

Multiplication. This emerges at larger scales than addition, consistent with multiplication being computationally harder. Models must implicitly implement the schoolbook algorithm or discover shortcuts.

Modular arithmetic. Operations like computing remainders show similar emergence patterns, appearing at scales where the model can track multiple computational steps.

Let's examine real scaling behavior on an arithmetic task:

The pattern is clear: models below ~10B parameters perform at or near random, while the 175B parameter model reaches 78% accuracy. This is not a gradual improvement; it's a phase transition.

Beyond arithmetic, several language understanding capabilities show emergent behavior:

• Word sense disambiguation. Determining which meaning of a polysemous word applies in context. Small models often default to the most common sense, while larger models correctly identify context-appropriate meanings.

• Coreference resolution. Tracking which pronouns refer to which entities across long passages. This requires maintaining and updating a mental model of discourse participants.

• Pragmatic inference. Understanding implied meaning beyond literal content. For example, recognizing that "Can you pass the salt?" is a request, not a question about ability.

Perhaps the most surprising emergent capabilities involve multi-step reasoning:

• Chain-of-thought reasoning. As we'll explore in Chapter 3, the ability to improve performance by generating intermediate reasoning steps emerges at scale. Smaller models do not benefit from step-by-step prompting; larger models do.

• Logical deduction. Following syllogistic reasoning or multi-step logical arguments. Small models fail at even simple two-step deductions that humans find trivial.

• Analogical reasoning. Transferring knowledge from one domain to another by recognizing structural similarities.

The ability to learn new tasks from examples provided in the prompt, which we discussed in the context of GPT-3 (Part XXVIII, Chapter 4), is itself an emergent capability. We'll examine this in detail in the next chapter.