Nucleus Sampling › Summary

Nucleus sampling addresses a fundamental limitation of top-k sampling by adapting the number of candidate tokens to the probability distribution at each step. Rather than asking "how many tokens should I consider?", nucleus sampling asks "how much probability mass should I keep?", and this reframing produces more consistent generation quality across varying contexts.

The core innovation is the shift from a count-based criterion to a probability-mass-based criterion. This single change enables the nucleus to contract when the model is confident and expand when the model is uncertain, tracking the actual structure of the distribution rather than imposing a fixed structure on it. Combined with temperature scaling, nucleus sampling gives practitioners two orthogonal controls over generation behavior: one for shaping the distribution and one for determining how much of it to retain.

The key ideas are:

• Cumulative probability threshold: The nucleus is the smallest token set whose probabilities sum to at least $p$
• Adaptive truncation: High-confidence predictions yield small nuclei; uncertain predictions yield large nuclei
• The unreliable tail: Tokens below the nucleus threshold are statistically noise that degrade generation quality when sampled
• Typical values: $p = 0.9$ to $0.95$ for general creative generation
• Combines with temperature: Temperature reshapes the distribution; nucleus sampling truncates it
• Practical default: Nucleus sampling is the most common choice for open-ended generation in modern systems

When implementing text generation, nucleus sampling should be your default starting point. Adjust $p$ based on your tolerance for creativity versus coherence, and combine with temperature scaling for finer control over the output distribution. Start at $p = 0.9$ with temperature $= 1.0$, then tune from there: lower $p$ or lower temperature for more conservative outputs, higher values for more creative ones.