Nucleus Sampling › Choosing the Right p Value

The probability threshold $p$ controls the trade-off between creativity and coherence, and choosing it well matters more than it might initially seem. Unlike top-k's $k$, which is difficult to reason about in the abstract, $p$ has a clear semantic meaning: it is the fraction of the probability distribution you want to sample from. Higher $p$ means you're including more of the tail, which produces more varied but potentially less coherent text. Lower $p$ means you're sampling from a more concentrated distribution, producing coherent but possibly repetitive or predictable text.

Here are practical guidelines for selecting $p$:

• $p = 0.9$ to $0.95$: General-purpose creative writing. This is the most common setting and works well for open-ended writing tasks, including story and dialogue generation. The nucleus is large enough for variety but excludes the unreliable tail.

• $p = 0.7$ to $0.9$: More focused generation. Useful when you want coherent output with moderate diversity, such as paraphrasing or generating multiple options for the same intent.

• $p < 0.7$: Very focused generation. Approaches greedy decoding behavior. Use for tasks requiring high precision, like code completion or factual responses.

• $p > 0.95$: Very permissive. Rarely used alone because it includes many low-probability tokens. Can produce surprising but occasionally incoherent outputs.

The default of $p = 0.9$ has held up surprisingly well across tasks and model sizes. Holtzman et al.'s original paper used this value in their experiments, and subsequent work has largely confirmed it as a useful starting point. The intuition is that 90% of the probability mass captures virtually all the plausible options while leaving out the long tail of noise.

In practice, the right $p$ can depend on the domain and the model's calibration. A model that is well-calibrated for a specific domain, such as a fine-tuned legal document generator, may benefit from lower $p$ because the distribution is already focused on domain-appropriate vocabulary. A general-purpose model generating creative fiction may benefit from higher $p$ to encourage novelty. The best approach is to generate several passages at candidate values of $p$ and evaluate them qualitatively, since automated metrics often fail to capture the trade-off between diversity and quality.

The value of $p$ interacts with the model's overall uncertainty, which changes over the course of a sequence. The beginning of a generation tends to have more uncertainty (the model hasn't established context yet), while later tokens tend to be more constrained by prior choices. This means the effective nucleus size varies naturally across a sequence even with a fixed $p$. In the early tokens, $p = 0.9$ might yield a nucleus of 50 tokens; in later tokens constrained by prior context, it might yield a nucleus of just 3. This dynamic adaptation is one of nucleus sampling's key strengths.

Let's visualize how $p$ affects generation diversity: