Decoding Temperature › Text Generation with a Real Model
Language AI HandbookAll chapters Decoding Temperature › Text Generation with a Real ModelOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's put temperature into practice with a complete text generation example using GPT-2. This demonstrates how temperature affects actual model output, not just abstract probability distributions. The key difference from our toy examples is that a real model generates tokens autoregressively: each sampled token is appended to the context, and the model predicts the next token conditioned on everything that came before. Temperature errors accumulate: an unexpected token early in generation can push the model into a context it was not trained on, leading to compounding degradation.
The autoregressive nature of generation also means that temperature has a cascading effect. When you select a less-probable token at position 5, you change the context for all subsequent positions. The model's logits at position 6 are now conditioned on that unusual choice, and those logits may themselves be unusual (because position 6 rarely followed that unusual token in training data). Over many steps at high temperature, this cascading can produce text that drifts far from coherent human language patterns. This is why the "error accumulation" concern is so important for long sequences.
from transformers import GPT2LMHeadModel, GPT2Tokenizer
# Load model and tokenizer
model_name = "gpt2"
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
model = GPT2LMHeadModel.from_pretrained(model_name, _fast_init=False)
model.eval()
# Set pad token to eos token (GPT-2 doesn't have a pad token by default)
tokenizer.pad_token = tokenizer.eos_tokendef generate_with_temperature(
prompt: str,
temperature: float,
max_new_tokens: int = 30,
model=model,
tokenizer=tokenizer,
) -> str:
"""Generate text completion with specified temperature using manual decoding."""
input_ids = tokenizer.encode(prompt, return_tensors="pt")
generated = input_ids.clone()
with torch.no_grad():
for _ in range(max_new_tokens):
outputs = model(generated)
next_token_logits = outputs.logits[:, -1, :]
if temperature <= 0:
# Greedy decoding
next_token = next_token_logits.argmax(dim=-1, keepdim=True)
else:
# Temperature sampling
scaled_logits = next_token_logits / temperature
probs = F.softmax(scaled_logits, dim=-1)
next_token = torch.multinomial(probs, num_samples=1)
generated = torch.cat([generated, next_token], dim=-1)
# Stop at EOS token
if next_token.item() == tokenizer.eos_token_id:
break
return tokenizer.decode(generated[0], skip_special_tokens=True)The generation loop mirrors exactly what we computed in the worked example above, but now operating on GPT-2's full 50,257-token vocabulary at each step. The model processes all generated tokens as context (including the newly added ones) on each iteration, which is why this is called autoregressive generation. Let's generate completions for a prompt at different temperatures:
Prompt: "The future of artificial intelligence is" ============================================================ Temperature = 0.3: ----------------------------------------
in the hands of the next generation of AI. The future of Temperature = 0.7: ----------------------------------------
uncertain, but there's much to like about the prospects. Explore Temperature = 1.0: ----------------------------------------
nowhere near as ambitious. Technological advances have few big promises. So the Temperature = 1.5: ----------------------------------------
world-changing how we all think—take foregone Great Fusion operators Heaven
At , the model produces focused, predictable continuations that sound like confident editorial statements. At , we see more varied vocabulary while maintaining coherence. At , creativity increases but the text may occasionally take unexpected turns in theme or word choice.
Comparing Multiple Samples
One sample does not reveal the full picture. A single draw at might happen to produce a diverse output, and a single draw at might happen to be conservative. The distribution of outputs across many draws is what temperature controls. Let's generate multiple completions at each temperature to see the range of outputs:
def generate_multiple(
prompt: str,
temperature: float,
n_samples: int = 5,
max_new_tokens: int = 20,
) -> list[str]:
"""Generate multiple completions to observe variety."""
completions = []
for _ in range(n_samples):
text = generate_with_temperature(prompt, temperature, max_new_tokens)
# Extract just the generated portion
generated = text[len(prompt) :].strip()
completions.append(generated)
return completionsPrompt: "In a world where robots" Temperature = 0.5: --------------------------------------------------
1. are ubiquitous and the Internet has grown exponentially, it 2. are becoming more and more ubiquitous, it's hard Temperature = 1.0: --------------------------------------------------
1. hunt everything in and at other times, they can 2. abound, one interesting trait that no One has now
At , the samples likely share similar themes and phrasing. At , you will observe more divergent narratives and word choices. This reflects the core promise of temperature: the same prompt can produce different completions depending on how much freedom you give the model to explore.
Measuring Output Diversity
We can quantify diversity by measuring how different the generated samples are from each other. One simple metric is the number of unique n-grams across samples. An n-gram is a sequence of consecutive words; unique n-grams across multiple samples indicate that the model is creating lexically diverse output rather than repeating the same phrases:
def measure_diversity(samples: list[str], n: int = 2) -> dict:
"""Measure n-gram diversity across samples."""
all_ngrams = []
for sample in samples:
words = sample.lower().split()
ngrams = [tuple(words[i : i + n]) for i in range(len(words) - n + 1)]
all_ngrams.extend(ngrams)
total = len(all_ngrams)
unique = len(set(all_ngrams))
return {
"total_ngrams": total,
"unique_ngrams": unique,
"diversity_ratio": unique / total if total > 0 else 0,
}Diversity comparison (10 samples, bigrams): Temperature Total Unique Diversity ------------------------------------------
0.3 27 24 0.889
0.7 21 21 1.000
1.0 20 20 1.000
1.5 23 23 1.000
Higher temperatures produce higher diversity ratios, confirming that the outputs explore a broader range of vocabulary and phrases. The diversity ratio measures what fraction of all n-grams across the samples are unique: a ratio of 1.0 would mean every bigram appeared exactly once (maximum diversity), while a ratio near 0 would indicate that all samples were nearly identical.