GPT-2 › Zero-Shot Task Performance
Language AI HandbookAll chapters GPT-2 › Zero-Shot Task PerformanceOne section · Sections in this chapter5 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
GPT-2's most surprising contribution was showing zero-shot task performance. Without any task-specific fine-tuning, GPT-2 could perform reading comprehension and summarization, as well as translation and question answering, simply by framing these tasks as language modeling. This was not a minor improvement on existing methods; it represented a fundamentally different relationship between a model and the tasks it could perform.
Before GPT-2, the standard workflow in NLP was: pretrain a model on a large corpus, then fine-tune it on labeled examples for each downstream task. Each task required its own fine-tuned model, its own training data, and its own deployment. GPT-2 suggested an alternative: one model, prompted appropriately, could attempt any task. The quality was not always competitive with fine-tuned specialists, but the existence of the capability at all was striking.
The Zero-Shot Paradigm
The key insight was that many NLP tasks can be expressed as conditional text generation. Rather than training separate models for each task, you can prompt a language model with a description of the task:
- Question answering: "Q: What is the capital of France? A:"
- Summarization: "Article: [article text] TL;DR:"
- Translation: "English: Hello, how are you? French:"
The language model, having seen similar patterns in its training data, learns to complete these prompts appropriately.
# Demonstrate zero-shot prompting patterns
zero_shot_examples = {
"reading_comprehension": """
Passage: The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars
in Paris, France. It is named after the engineer Gustave Eiffel, whose company
designed and built the tower.
Question: Who designed the Eiffel Tower?
Answer:""",
"summarization": """
Article: Scientists have discovered a new species of deep-sea fish in the
Mariana Trench. The fish, nicknamed the "ghost fish" due to its translucent
appearance, was found at depths exceeding 8,000 meters. Researchers believe
the fish has adapted unique biological mechanisms to survive the extreme
pressure at such depths.
TL;DR:""",
"translation": """
English: The weather is beautiful today.
French:""",
"sentiment": """
Review: This movie was absolutely incredible! The acting was superb and the
plot kept me on the edge of my seat the entire time.
Sentiment:""",
}Zero-Shot Prompt Examples: ============================================================ READING COMPREHENSION: ---------------------------------------- Passage: The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower. Question: Who designed the Eiffel Tower? Answer: SUMMARIZATION: ---------------------------------------- Article: Scientists have discovered a new species of deep-sea fish in the Mariana Trench. The fish, nicknamed the "ghost fish" due to its translucent appearance, was found at depths exceeding 8,000 meters. Researchers believe the fish has adapted unique biological mechanisms to survive the extreme pressure at such depths. TL;DR: TRANSLATION: ---------------------------------------- English: The weather is beautiful today. French: SENTIMENT: ---------------------------------------- Review: This movie was absolutely incredible! The acting was superb and the plot kept me on the edge of my seat the entire time. Sentiment:
Each prompt demonstrates a different task expressed as conditional text generation. The reading comprehension prompt provides context then asks a question. The summarization prompt uses "TL;DR:" as a signal the model learned from internet forums. The translation prompt establishes a bilingual pattern the model should continue. The sentiment prompt gives a review and expects the model to complete the word that naturally follows. These formats work because WebText contained millions of similar patterns, each teaching the model what kind of text should complete each kind of prefix.
Benchmark Results
OpenAI evaluated GPT-2 on various benchmarks in zero-shot mode, comparing against supervised baselines that were explicitly trained for each task. Perplexity is the primary metric: it measures how surprised the model is by each word in the test text. Lower perplexity means the model assigns higher probability to the correct next word. A perfect model would have perplexity of 1 (never surprised); a random model over a 50,000-word vocabulary would have perplexity around 50,000.
The relationship between model size and perplexity follows a consistent pattern across benchmarks. Let's examine this scaling behavior more closely:
On a log-log plot, the near-linear relationship between parameters and perplexity suggests power-law scaling. This empirical observation, that performance improves predictably with scale, would later be formalized in the "Scaling Laws for Neural Language Models" paper by Kaplan et al. (2020). The consistency across diverse benchmarks indicates that scaling benefits are not dataset-specific but reflect improvements in language modeling capability. The key insight is that perplexity follows a power law with parameter count, meaning you can predict in advance how much a model will improve if you double its size.
Several patterns emerged from these evaluations:
-
Consistent scaling: Larger models achieved lower perplexity across all benchmarks. GPT-2 XL achieved state-of-the-art on Penn Treebank despite never being explicitly trained on it.
-
Domain transfer: Performance on WikiText (derived from Wikipedia) was strong despite Wikipedia being excluded from training, suggesting the model learned transferable language patterns.
-
Task emergence: Capabilities like reading comprehension and translation emerged without task-specific training, purely from scale and diverse pretraining.
Limitations of Zero-Shot
Zero-shot performance, while impressive, came with clear limitations:
-
Inconsistency: The model sometimes generated plausible-sounding but incorrect answers, particularly for factual questions.
-
Sensitivity to prompting: Small changes in prompt format could significantly affect performance.
-
Task ambiguity: Without examples, the model sometimes misinterpreted what was being asked.
These limitations motivated the subsequent exploration of few-shot learning, where giving a handful of examples dramatically improved task performance. GPT-3 would show that giving just 10 to 20 examples in the prompt could match or exceed the performance of fine-tuned models trained on thousands of examples. The limitations of pure zero-shot learning thus look less like a basic barrier and more like a problem of insufficient context.