Nucleus Sampling › Decoding Strategies

Nucleus sampling is one of several strategies for decoding from a language model's probability distribution, and understanding how it fits among the alternatives helps you choose the right tool for each task.

Greedy decoding always selects the single highest-probability token at each step. It is fast and deterministic, but it is prone to repetition and produces text that feels mechanical. The problem with greedy decoding is that locally optimal choices are not globally optimal: choosing the most probable token at step $t$ can close off better paths at steps $t+1$ and beyond.

Beam search addresses this by maintaining multiple candidate sequences simultaneously, expanding each by one token at a time and keeping the top-$b$ sequences by cumulative log-probability. This produces more globally coherent completions than greedy decoding, but it tends to generate safe, generic text that lacks the diversity and naturalness of human writing. Beam search also has failure modes in long-form generation, where it produces degenerate repetitive sequences.

Pure sampling draws a token from the full probability distribution without any truncation. This maximizes diversity but regularly samples from the tail, producing incoherent text over longer generations.

Temperature scaling combined with truncation (top-k or top-p) occupies a middle ground. The truncation eliminates the harmful tail while preserving diversity within the plausible region. Nucleus sampling represents the principled version of this middle ground because it defines "plausible" in terms of probability mass rather than token count.

More recently, researchers have proposed contrastive search, typical sampling, and other variants that address different failure modes. Contrastive search penalizes repetition by comparing generated text against the context. Typical sampling targets the "typical" probability region of the distribution rather than the highest-probability tokens. These methods are complementary to nucleus sampling and can sometimes be combined with it.

For most practical applications, nucleus sampling at $p = 0.9$ combined with moderate temperature remains the standard starting point. The alternatives are worth exploring when you have specific failure modes to address.