Nucleus Sampling › Nucleus Sampling vs. Top-k: When to Use Which

Both methods truncate the vocabulary, but they do so with different philosophies:

In practice, nucleus sampling is preferred for creative text generation because of its adaptive behavior. Top-k remains useful when you want explicit control over the number of options or when slight computational savings matter at scale.

Some systems use both: first apply top-k as a coarse filter (e.g., k=100), then apply nucleus sampling within that set. This combines the efficiency of top-k with the adaptivity of top-p. The top-k pass quickly eliminates the very bottom of the vocabulary without needing a cumulative sum, and the subsequent nucleus pass makes the fine-grained probability-based cutoff within the reduced candidate set. For very large vocabularies (100,000+ tokens), this two-pass approach can reduce computation noticeably.

Another consideration is reproducibility. Top-k is deterministic in its selection set: the same distribution always produces the same $k$ candidates. Nucleus sampling is also deterministic in selection but can include different numbers of tokens across calls with the same $p$, since the nucleus size is data-dependent. This flexibility is usually an advantage, but for debugging purposes, it helps to know that the nucleus size will vary across steps.