Emergence vs Metrics › The Measurement Problem

When we evaluate language models on tasks, we face a basic choice: how do we convert continuous model outputs into discrete success/failure judgments? This choice has measurable implications for how capabilities appear to develop.

Think of a language model as a machine that assigns probabilities to every token in its vocabulary at each position in a sequence. When answering a multiple-choice question, the model doesn't internally choose "A." It assigns probability mass across all possible continuations, and we read off which choice received the most probability. The model's internal state contains a rich distribution of confidence, but the moment we ask "did it get the question right?" we collapse that entire distribution into a single bit of information: 0 or 1, wrong or right.

Consider a multiple-choice question with four options. A model produces probability distributions over tokens, which we interpret as confidence in each answer choice. But benchmarks don't report these probabilities directly; they report whether the model got the answer right or wrong. This binary reduction throws away most of the information about what the model learned.

To see how severe this information loss is, consider a student taking a test. One student confidently writes the correct answer with complete certainty. Another student narrows down to two choices and makes an educated guess that happens to be right. A third student was 90% sure of the correct answer but second-guessed themselves at the last moment. Traditional grading treats all three identically: all three receive full marks. Yet clearly these students have different levels of understanding. The same information collapse occurs when we evaluate language models using binary accuracy metrics.

The key insight is that this information loss is not uniformly distributed across scale. Smaller models tend to hover near the decision boundary, where accuracy discards the most information. As models improve through scaling, they push their probabilities progressively farther from the boundary, but accuracy only "notices" this improvement once the boundary is crossed. This asymmetry means that accuracy metrics systematically understate early capability gains and then suddenly "catch up" when probabilities finally exceed 50%, creating the illusion of discontinuous improvement where none exists.

The metrics we use for evaluation fall into two broad categories. Continuous metrics like cross-entropy loss, Brier score, or token-level log-probabilities can reveal gradual improvements. Discontinuous metrics like accuracy, exact match, or pass@1 collapse this information into binary outcomes.

To understand the difference concretely: if a model improves its confidence in the correct answer from 30% to 49%, a continuous metric like Brier score captures this substantial improvement, while accuracy still reports zero, since the model was "wrong" both times. The model has learned something valuable, becoming nearly twice as confident in the correct answer, yet our measurement instrument is blind to this progress.

Most emergence papers relied on discontinuous metrics. The sudden appearance of capabilities they documented may tell us more about the metrics than about the models.