Nucleus Sampling › The Unreliable Tail
Language AI HandbookAll chapters Nucleus Sampling › The Unreliable TailOne section · Sections in this chapter4 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To appreciate why nucleus sampling matters, it helps to understand what exactly lives in the tail of a language model's output distribution. The tail contains tokens that are much less likely than the leading candidates. In the current context, a well-calibrated model should assign these tokens near-zero probability. The fact that they receive small but non-zero probabilities reflects the imperfection of current models.
Holtzman et al. published "The Curious Case of Neural Text Degeneration" in 2020, and the paper's central finding was striking. They showed empirically that neural language models, despite achieving low perplexity, could generate text that degraded rapidly in quality. The culprit was the unreliable tail: over long sequences, sampling from the full distribution repeatedly landed on low-probability tokens, and each such error compounded the next. They coined the term "neural text degeneration" and proposed nucleus sampling as the cure. The paper immediately influenced how practitioners thought about generation quality and established the standard that nucleus sampling should be the default for open-ended generation.
Language models learn to assign small probabilities to nearly everything in the vocabulary, because neural networks with softmax outputs are rarely completely certain. Even when "Paris" has 95% probability, the remaining 5% is spread across thousands of tokens: "Pari", "paris", "PARIS", "Lyon", and also nonsense like "the", "xyz", "purple", and every other token in the vocabulary. The tail is mostly nonsense, with occasional legitimate alternatives mixed in.
The problem is that when you have 50,000 tokens each with 0.001% probability, sampling from all of them occasionally picks something terrible. In practice, over a 200-token generation with a large vocabulary, you might accidentally sample from the unreliable tail several times. Each such accident corrupts the context for subsequent tokens, often in ways that are hard to recover from. A sentence that starts generating well can suddenly produce a strange word, and subsequent tokens try to make sense of the strange word, leading to increasingly incoherent text.
Top-k sampling with a large mitigates this by excluding the very bottom of the tail, but it still includes tokens with probabilities like 0.01% when . Nucleus sampling is more principled: by defining the nucleus as 90% of the total mass, it excludes everything below a contextually-determined probability floor. In high-certainty contexts, that floor is high (only a few tokens qualify). In uncertain contexts, the floor is lower (many tokens qualify), but the floor always reflects the actual distribution shape.
In practice, the safest way to think about the tail is this: any token with probability low enough to be excluded by nucleus sampling is a token the model is, in effect, saying should not be generated here. Forcing it into the sampling pool overrides the model's own judgment. Nucleus sampling respects what the model learned.