Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GPT-2 › Generation Quality

Michael BrenndoerferUpdated July 24, 20258 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 198GPT-2 › Generation Quality

Reading section 6 of 11 · Generation Quality · 8 minSection 6 of 11 · 8 min

Section 6 of 11 · 8 min read

GPT-2 › Generation QualityOne section · Sections in this chapter6 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

GPT-2's text generation quality captured public attention more than its benchmark scores. The model could produce coherent, contextually appropriate text that often fooled readers into thinking it was human-written. This section examines the technical mechanisms behind that quality and the parameters that control the tradeoff between coherence and creativity.

Coherent Long-Form GenerationLink Copied

Unlike previous language models that quickly devolved into nonsense, GPT-2 maintained coherence over paragraphs and pages. The 1,024-token context window was part of this: the model could remember the topic and characters, along with constraints introduced earlier in a passage. But the transformer's attention mechanism was equally important. Unlike RNN-based models that encoded context into a fixed-size hidden state, the transformer can directly attend to any position in the context window, maintaining specific details without them degrading through recurrent state updates.

Think of the difference this way. An RNN reading a 1,000-word passage must compress everything it reads into a fixed-size vector, the way you might try to remember a long lecture using only five index cards. The transformer, by contrast, keeps the entire lecture transcript available and can look back at any part of it when generating the next word. The 1,024-token context means GPT-2 keeps roughly 750 words of context fully accessible for every generation step.

Let's examine the sampling strategies that enable quality generation:

In[19]:

Code

def top_k_sampling(
    logits: torch.Tensor, k: int = 50, temperature: float = 1.0
) -> int:
    """Sample from top-k logits with temperature scaling."""
    # Apply temperature
    logits = logits / temperature

    # Get top-k logits and indices
    top_k_logits, top_k_indices = torch.topk(logits, k)

    # Convert to probabilities
    probs = F.softmax(top_k_logits, dim=-1)

    # Sample from the distribution
    sampled_idx = torch.multinomial(probs, num_samples=1)

    return top_k_indices[sampled_idx].item()


def nucleus_sampling(
    logits: torch.Tensor, p: float = 0.9, temperature: float = 1.0
) -> int:
    """Sample from nucleus (top-p) distribution."""
    # Apply temperature
    logits = logits / temperature

    # Sort logits and get cumulative probabilities
    sorted_logits, sorted_indices = torch.sort(logits, descending=True)
    cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)

    # Remove tokens outside the nucleus
    sorted_indices_to_remove = cumulative_probs > p
    # Keep at least one token
    sorted_indices_to_remove[0] = False

    # Set removed logits to -inf
    sorted_logits[sorted_indices_to_remove] = float("-inf")

    # Convert to probabilities and sample
    probs = F.softmax(sorted_logits, dim=-1)
    sampled_idx = torch.multinomial(probs, num_samples=1)

    return sorted_indices[sampled_idx].item()

The quality of generated text depends heavily on the sampling strategy. Pure random sampling from the full distribution produces incoherent text because low-probability tokens occasionally get selected. Consider a case where the model is 90% sure the next word is "the" but random sampling occasionally selects one of the 49,000 other tokens. Even a 1% chance of selecting a bizarre word means that on average once per hundred tokens, the text swerves in an unexpected direction. Over a 500-word passage, this guarantees several incoherent intrusions. Top-k and nucleus sampling restrict generation to high-probability tokens, dramatically improving coherence.

Out[20]:

Visualization

Bar chart showing top-k sampling selecting the 5 highest probability tokens. — Top-k sampling keeps the k highest probability tokens.

Bar chart showing nucleus sampling selecting tokens until cumulative probability reaches 0.9. — Nucleus (top-p) sampling keeps tokens until cumulative probability exceeds p.

Temperature and CreativityLink Copied

The temperature parameter provides a tunable tradeoff between coherence and creativity. Temperature scales the logits before applying softmax, controlling how "peaked" or "flat" the resulting probability distribution becomes.

The temperature-scaled softmax computes the probability of selecting token $i$ as:

P(x_i) = \frac{\exp(z_i / T)}{\sum_{j=1}^{V} \exp(z_j / T)}

where:

$P(x_i)$ : the probability of selecting token $i$
$z_i$ : the raw logit (unnormalized score) for token $i$ from the model's output layer
$T$ : the temperature parameter, a positive scalar
$V$ : the vocabulary size (total number of possible tokens)
$\exp(\cdot)$ : the exponential function

When $T = 1$ , this reduces to standard softmax. As $T \to 0$ , the distribution becomes increasingly peaked around the maximum logit (approaching argmax). As $T \to \infty$ , the distribution approaches uniform, giving all tokens equal probability regardless of their logits. This happens because dividing by a large $T$ compresses all logit differences toward zero.

Why does this formula make sense? Notice that dividing each logit by $T$ rescales the differences between logits before softmax is applied. When $T = 0.5$ , a logit difference of 2 becomes a difference of 4, making the leading token much more dominant. When $T = 2$ , a logit difference of 2 becomes only 1, compressing the probability gap between tokens. Temperature is thus a direct control on how decisive the model is: low temperature makes it more deterministic and conservative, high temperature makes it more exploratory and potentially unpredictable.

Out[21]:

Visualization

Bar chart showing how probability distributions change across different temperature values. — Effect of temperature on probability distribution. Lower temperatures concentrate probability on the top token, while higher temperatures approach uniform distribution.

At temperature 0.3, the model almost always selects the highest-probability token, creating repetitive but grammatically correct text. At temperature 1.0 (the default), it balances variety with coherence. At temperature 2.0, even low-probability tokens have significant chance of selection, leading to creative but potentially nonsensical output. For practical applications: use low temperatures (0.3 to 0.7) for factual question answering where accuracy matters, and higher temperatures (0.9 to 1.2) for creative writing or brainstorming where diversity is valuable.

Sample GenerationLink Copied

Let's demonstrate generation using a pretrained GPT-2 model from Hugging Face:

In[8]:

Code

from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load pretrained GPT-2 (smallest model for efficiency)
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
model.eval()


def generate_text(
    prompt: str,
    max_new_tokens: int = 50,
    temperature: float = 0.8,
    top_p: float = 0.9,
) -> str:
    """Generate text continuation using GPT-2."""
    input_ids = tokenizer.encode(prompt, return_tensors="pt")

    with torch.no_grad():
        output_ids = model.generate(
            input_ids,
            max_new_tokens=max_new_tokens,
            temperature=temperature,
            top_p=top_p,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )

    return tokenizer.decode(output_ids[0], skip_special_tokens=True)

The generated text demonstrates GPT-2's ability to maintain topical coherence and grammatical correctness. It continues prompts in contextually appropriate ways, though close inspection often reveals subtle errors in logic or factual accuracy. The first prompt tends to produce technology commentary that sounds authoritative but may conflate different AI approaches. The creative writing prompt typically produces flowing narrative prose. The science prompt often produces plausible-sounding but fabricated findings. These patterns are characteristic of GPT-2: impressively coherent surface structure, unreliable deep content.