Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Emergence vs Metrics › Threshold Effects in Accuracy Metrics

Michael BrenndoerferNovember 8, 20259 min read

Language AI HandbookAll chapters

Part 30 of 67Emergent Capabilities

Chapter 217Emergence vs Metrics › Threshold Effects in Accuracy Metrics

Reading section 3 of 13 · Threshold Effects in Accuracy Metrics · 9 minSection 3 of 13 · 9 min

Section 3 of 13 · 9 min read

Emergence vs Metrics › Threshold Effects in Accuracy MetricsOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To understand why discontinuous metrics create apparent emergence, we examine the mathematics of threshold effects. The core insight is that when we demand perfect sequential correctness, we transform what might be gradual improvement into an all-or-nothing proposition.

Consider a task where a model must generate an exact string like "The answer is 42." The model produces this string autoregressively, with some probability $p_i$ of generating the correct token at position $i$ given all previous tokens were generated correctly. Each token represents a checkpoint that the model must pass. Failing at any point means failing entirely.

The probability of generating the entire correct sequence is:

P(\text{correct sequence}) = \prod_{i=1}^{n} p_i

where:

$P(\text{correct sequence})$ : the probability of generating the entire target string correctly
$p_i$ : the probability of generating the $i$ -th correct token, given all previous tokens were correct
$n$ : the total number of tokens in the target sequence
$\prod_{i=1}^{n}$ : the product operator over all $n$ tokens, applying the chain rule of probability for sequential generation, where each token's probability is conditioned on all previous tokens being correct

This formula shows that sequential correctness is fragile. Unlike addition, where errors might partially cancel, multiplying probabilities means every imperfect step compounds the risk of failure. A chain is only as strong as its weakest link, and here we're multiplying all the links together. Think of it like a relay race where each runner must complete their leg perfectly. One stumble anywhere and the entire team loses, regardless of how well everyone else performed.

This multiplicative relationship is important: even small imperfections compound. If each token has a 95% chance of being correct, the probability of getting all tokens right shrinks rapidly as the sequence grows longer. What seems like excellent per-step performance (after all, 95% accuracy sounds impressive) leads to surprisingly poor overall outcomes when steps must chain together.

If we assume roughly uniform per-token accuracy $p$ across all $n$ tokens, this becomes:

P(\text{exact match}) = p^n

This exponential relationship creates a threshold effect. The exponent $n$ acts as an amplifier, transforming modest changes in $p$ into dramatic changes in overall success probability. Suppose the model's per-token accuracy improves linearly with log compute from 80% to 99% over several orders of magnitude of scaling. For a 10-token answer:

Sequence accuracy as a function of per-token accuracy for a 10-token sequence.
Per-token accuracy	Sequence accuracy
80%	10.7%
85%	19.7%
90%	34.9%
95%	59.9%
99%	90.4%

The per-token accuracy improved smoothly from 80% to 99%, but sequence accuracy appears to "emerge" around the 95% per-token threshold. A model with 80% per-token accuracy looks like it has nearly zero capability on this task, but it is closer to succeeding than the metric suggests. This table shows a clear asymmetry: the jump from 95% to 99% per-token accuracy (a mere 4 percentage points) produces roughly the same sequence accuracy improvement as the entire journey from 80% to 95% (15 percentage points). The mathematics of multiplication creates this hockey-stick pattern automatically.

Out[3]:

Visualization

Heatmap showing sequence accuracy as a function of per-token accuracy and sequence length, with contour lines marking key probability thresholds. — Heatmap showing sequence accuracy as a function of per-token accuracy and sequence length. The color transition from yellow to red reveals the emergence threshold zone where small improvements in per-token performance produce dramatic gains in sequence accuracy. Contour lines mark key probability levels, showing how longer sequences require increasingly higher per-token accuracy to achieve the same overall performance.

The heatmap reveals the "emergence zone" where sequence accuracy transitions rapidly from near-zero (red) to high performance (green). Notice how this transition zone shifts rightward for longer sequences. A 50-token task requires over 95% per-token accuracy to achieve even 50% sequence accuracy, while a 10-token task reaches the same performance at around 93% per-token accuracy.

The Amplification ProblemLink Copied

Longer sequences amplify this effect dramatically, turning modest measurement issues into severe distortions of apparent capability. The mathematical relationship between sequence length and success probability follows an inexorable exponential decay. For a 50-token sequence:

P(\text{exact match}) = 0.95^{50} \approx 0.077

where:

$0.95$ : the per-token accuracy (95% chance of getting each token correct)
$50$ : the number of tokens in the target sequence
$0.95^{50}$ : the result of multiplying 0.95 by itself 50 times
$\approx 0.077$ : approximately 7.7% probability of generating the entire sequence correctly

To see why this happens, consider that $0.95^{50}$ means multiplying 0.95 by itself 50 times. Each multiplication slightly shrinks the result: $0.95 \times 0.95 = 0.9025$ , then $0.9025 \times 0.95 = 0.857$ , and so on. After 50 such multiplications, even starting from 95% per step, the cumulative probability has decayed to under 8%. This decay is relentless and mechanical. No amount of hoping or clever prompting can overcome the basic mathematics of multiplied probabilities.

Even a 95% per-token model achieves less than 8% exact match on a 50-token sequence. This model has substantial capability, but the exact match metric makes it appear completely incompetent. Consider a surgeon who performs each step of a 50-step procedure with 95% success. We would consider them highly skilled. Yet our evaluation framework would label their overall performance as a failure more than 92% of the time.

The relationship between per-token probability and sequence accuracy follows the exponential formula $P = p^n$ , where small changes in $p$ produce dramatic changes in $P$ for large $n$ . This mathematical structure is not a bug in our analysis; it is an accurate description of how exact-match evaluation works. The question is whether this evaluation accurately reflects the capability we care about, or whether it creates artificial cliffs that obscure underlying progress.

Out[4]:

Visualization

Line plot showing sequence accuracy decay curves for different per-token accuracies across increasing sequence lengths, with a horizontal dashed line marking the 50% success threshold. — Probability decay curves showing how sequence accuracy declines with increasing sequence length for different per-token accuracies. Even excellent per-token performance of 95% decays rapidly as sequences grow longer, while the performance gap between 95% and 99% per-token accuracy becomes enormous for long sequences. The horizontal line marks the 50% success threshold, showing how longer sequences push capabilities below chance performance.

In[5]:

Code

import numpy as np


def sequence_accuracy(per_token_prob: float, sequence_length: int) -> float:
    """Calculate probability of generating entire sequence correctly."""
    return per_token_prob**sequence_length


# Demonstrate threshold effect for different sequence lengths
per_token_probs = np.linspace(0.80, 0.99, 20)
sequence_lengths = [5, 10, 25, 50]

results = {}
for length in sequence_lengths:
    results[length] = [sequence_accuracy(p, length) for p in per_token_probs]

Out[6]:

Console

Per-token prob Length 5 Length 10 Length 25 Length 50
           80%    32.8%     10.7%      0.4%      0.0%
           85%    44.4%     19.7%      1.7%      0.0%
           90%    59.0%     34.9%      7.2%      0.5%
           95%    77.4%     59.9%     27.7%      7.7%
           99%    95.1%     90.4%     77.8%     60.5%

Notice how longer sequences push the "emergence threshold" higher. A task requiring 50-token outputs won't show above-chance accuracy until per-token probability exceeds roughly 97%. This creates the illusion that long-form generation capabilities emerge suddenly at large scales. The capability didn't emerge suddenly. Our measurement tool simply wasn't sensitive enough to detect the gradual improvement happening underneath.

Worked Example: The Arithmetic Task

Consider a model asked to compute "17 + 28 = ?" and generate the token "45". This is a single-token output, so exact match simply checks whether the top prediction is correct. As the model scales, its probability on "45" might increase from 20% at small scale to 55% at medium scale. Accuracy would show this as an emergence: 0% at small scale, jumping to 100% at medium scale (assuming "45" is the argmax once probability exceeds 50%). But nothing discontinuous happened. The model's confidence in the correct answer climbed steadily.

Now consider a harder arithmetic task where the answer is "1,247". The model must generate four tokens sequentially: "1", ",", "2", "4", "7". If each token has 85% per-token accuracy, overall exact match is $0.85^5 = 0.44$ . At 90% per-token accuracy, it becomes $0.90^5 = 0.59$ . This small per-token improvement looks like a large accuracy jump. The four-digit arithmetic task "emerges" not because something basic changed, but because we asked for a five-token sequence instead of a one-token sequence. The task difficulty amplified the threshold effect.