Emergence in Neural Networks › Defining Emergence
Language AI HandbookAll chapters Emergence in Neural Networks › Defining EmergenceOne section · Sections in this chapter2 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The term "emergence" comes from complex systems theory, where it describes properties that arise from the interaction of simpler components but cannot be predicted from those components alone. A classic example is consciousness arising from neurons, or the flocking behavior of birds arising from simple local rules. In each case, the whole exhibits properties qualitatively different from its parts.
Emergence refers to capabilities that are not present in smaller models but appear in larger models, particularly when these capabilities appear abruptly rather than improving gradually with scale.
For language models, emergence typically means a capability that:
- Is absent at small scales: The model performs at or near random chance on the task
- Appears at larger scales: Performance suddenly jumps to above-chance levels
- Shows discontinuous improvement: The transition is sharp rather than gradual
This definition focuses on the scaling behavior of capabilities rather than their inherent nature. A task is not intrinsically emergent. Rather, emergence describes how performance on that task changes as we scale up models.
The formal study of emergence in LLMs was crystallized by Wei et al. (2022) in their paper "Emergent Abilities of Large Language Models." They defined emergent abilities as those that are "not present in smaller models but are present in larger models." They also operationalized "not present" as performance that is indistinguishable from random guessing.
Weak vs Strong Emergence
Philosophers distinguish between two types of emergence that are useful for thinking about LLMs:
Weak emergence refers to properties that are surprising or difficult to predict from lower-level descriptions but are in principle fully reducible to those descriptions. Given enough knowledge about the components and their interactions, we could derive the emergent property. Most cases of emergence in neural networks fall into this category: even if we cannot easily predict when a capability will appear, the capability ultimately results from the learned weights and computations.
Strong emergence refers to properties that are irreducible to lower-level descriptions. No amount of knowledge about the components would allow prediction of the emergent property. Strong emergence remains philosophically controversial, and there is no clear evidence that neural networks exhibit it.
For our purposes, we'll focus on weak emergence: capabilities that are surprising and appear discontinuously with scale, even if they're ultimately reducible to the network's computations.