Nucleus Sampling › Key Parameters

Michael BrenndoerferUpdated July 30, 20251 min read
Language AI HandbookAll chapters
Part 28 of 67GPT Architecture
Chapter 204Nucleus Sampling › Key Parameters
Reading section 12 of 14 · Key Parameters · 1 minSection 12 of 14 · 1 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 204 of 513

Section 12 of 14 · 1 min read

Nucleus Sampling › Key ParametersOne section · Sections in this chapter12 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When using nucleus sampling in Hugging Face's generate() method, the following parameters control the sampling behavior:

  • top_p (float, 0.0 to 1.0): The cumulative probability threshold. Only tokens whose cumulative probability mass is within this threshold are considered. Higher values include more tokens, increasing diversity. Typical range: 0.9 to 0.95 for creative tasks, 0.7 to 0.9 for more focused generation.

  • temperature (float, > 0): Scales the logits before applying softmax. Values below 1.0 sharpen the distribution (more deterministic), values above 1.0 flatten it (more random). Applied before nucleus truncation.

  • top_k (int): When using nucleus sampling alone, set to 0 to disable top-k filtering. Can be combined with top-p for hybrid filtering.

  • do_sample (bool): Must be set to True to enable any sampling strategy. When False, the model uses greedy decoding.

  • num_return_sequences (int): Number of independent completions to generate. Useful for generating multiple diverse options.