Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Nucleus Sampling › Combining Nucleus Sampling with Temperature

Michael BrenndoerferUpdated July 30, 20255 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 204Nucleus Sampling › Combining Nucleus Sampling with Temperature

Reading section 8 of 14 · Combining Nucleus Sampling with Temperature · 5 minSection 8 of 14 · 5 min

Section 8 of 14 · 5 min read

Nucleus Sampling › Combining Nucleus Sampling with TemperatureOne section · Sections in this chapter8 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Nucleus sampling and temperature scaling address different aspects of the probability distribution. Temperature reshapes the distribution, making it more peaked or more uniform. Nucleus sampling then truncates based on the reshaped distribution. Understanding how they interact helps you use both controls more precisely.

Temperature scaling modifies the logits before the softmax step. Dividing logits by a temperature $T < 1$ sharpens the distribution, pushing more mass to the top tokens. Dividing by $T > 1$ flattens it, spreading mass more evenly across the vocabulary. The nucleus threshold $p$ then determines how much of this reshaped distribution to keep.

Think of temperature as the "shape dial" and $p$ as the "inclusion dial". Temperature decides how concentrated the distribution is; $p$ decides what fraction of that distribution to retain. When you lower temperature, the model becomes more confident, so the nucleus naturally shrinks (even with the same $p$ , fewer tokens are needed to reach the threshold). When you raise temperature, the model spreads probability more widely, so the nucleus expands.

This interaction gives you two independent dimensions of control over generation behavior. You can hold $p$ fixed at 0.9 and vary temperature to change how deterministic the generation feels. Alternatively, you can hold temperature fixed and vary $p$ to change how cautiously you sample from the model's distribution. In practice, most systems expose both parameters and let users tune them independently, which provides the flexibility needed across diverse generation tasks.

Out[13]:

Visualization

Bar chart showing probability distribution at temperature 0.5 with nucleus of 1-2 tokens. — Temperature T=0.5 concentrates probability in the top token, shrinking the nucleus to just 1-2 tokens.

Bar chart showing probability distribution at temperature 1.0 with nucleus of 3 tokens. — Temperature T=1.0 (no scaling) shows the baseline distribution with a moderate nucleus size.

Bar chart showing probability distribution at temperature 1.5 with nucleus of 5 tokens. — Temperature T=1.5 spreads probability more evenly, expanding the nucleus to capture more tokens.

The visualization above shows how temperature and nucleus size interact. At low temperature (T=0.5), probability concentrates heavily in the top token, so the nucleus contains just 1-2 tokens. At high temperature (T=1.5), probability spreads more evenly, requiring 5+ tokens to capture 90% of the mass. This interaction lets you fine-tune generation behavior: temperature controls the shape of the distribution, while the nucleus threshold determines how much of that reshaped distribution you sample from.

Using both together gives you fine-grained control:

Low temperature + high p: Coherent output with some variety. The temperature concentrates probability mass, while high p still allows sampling from multiple tokens.
High temperature + low p: Unusual but still controlled. Temperature spreads probability, but low p keeps only the tokens that remain relatively high after spreading.
High temperature + high p: Maximum creativity. Use with caution as outputs can become incoherent.

In[14]:

Code

# Generate with combined temperature and nucleus sampling
with torch.no_grad():
    # Lower temperature for more focused but still diverse output
    focused_outputs = model.generate(
        input_ids,
        max_new_tokens=50,
        do_sample=True,
        top_p=0.9,
        temperature=0.7,  # Slightly lower temperature
        pad_token_id=tokenizer.eos_token_id,
        num_return_sequences=2,
    )

    # Higher temperature for more creative output
    creative_outputs = model.generate(
        input_ids,
        max_new_tokens=50,
        do_sample=True,
        top_p=0.95,
        temperature=1.2,  # Higher temperature
        pad_token_id=tokenizer.eos_token_id,
        num_return_sequences=2,
    )

Out[15]:

Console

Focused (temp=0.7, p=0.9):

  The future of artificial intelligence is in the hands of the future, but it may not be that way at the moment."

This is a quote from the paper "A new class of artificial intelligence is emerging that will be able to outperform human intelligence in tasks such as the

  The future of artificial intelligence is now in the hands of a group of engineers who have created a tool that will make it easier to predict the future.

Advertisement

The new software, called "Big Data," will be called "Machine Learning," and is expected to be


Creative (temp=1.2, p=0.95):

  The future of artificial intelligence is the world, in which every computer is an entirely different thing – and at least one of them will have an uncanny ability to pick up patterns. One of these "intelligence-enhanced" machines known as the "AI" will emerge to help humanity

  The future of artificial intelligence is looking brighter. It does it from a computational neuroscience and not through any of the usual "mechanics" techniques. But then there's machine learning, which is doing this sort of stuff on a large scale. It gives you an idea why something

The focused outputs with lower temperature tend to stay closer to common phrasings and high-probability continuations. The creative outputs with higher temperature explore more unusual word choices and directions, though they may occasionally produce less coherent passages. Finding the right balance depends on your application: conversational AI typically benefits from moderate settings, while creative writing tools can push toward higher values.