GPT-3 › Evaluating GPT-3's Capabilities

The GPT-3 paper evaluated the model across dozens of NLP benchmarks, from reading comprehension and question answering to commonsense reasoning and word analogy; it also covered arithmetic, translation, plus coding tasks. This breadth was deliberate: the authors wanted to demonstrate that few-shot learning was a general capability rather than a performance trick on a specific benchmark.

Performance varied dramatically by task type, revealing both the potential and limits of in-context learning. Some tasks saw GPT-3 few-shot performance that matched or exceeded fine-tuned state-of-the-art. Others saw GPT-3 underperform a simple majority-class baseline. Understanding this variance is essential for developing accurate intuitions about what large language models can and cannot do.

GPT-3 achieved impressive results on tasks requiring broad language understanding:

On TriviaQA, GPT-3's few-shot performance (71.2%) approached the fine-tuned state-of-the-art (75.4%), despite never being explicitly trained on the task. This is remarkable: models fine-tuned on TriviaQA have access to thousands of question-answer pairs during training, have optimized their weights specifically for this task, and still only barely outperform GPT-3 given a handful of examples. On physical commonsense reasoning (PIQA), GPT-3 exceeded the previous best fine-tuned model, suggesting that some commonsense knowledge is better learned from broad pretraining than from narrow task-specific datasets.

The word scrambling result is particularly striking because it suggests the model has internalized something about the structure of English words. Given a scrambled word like "lykaeh" and asked to find the unscrambled form, GPT-3 could often correctly identify "weakly" or similar targets. This is not a task most humans find trivial, and it was not explicitly part of any training objective. The model appears to have developed implicit knowledge of English phonology and orthography as a byproduct of predicting text.

Perhaps most striking were capabilities that emerged without explicit training:

• Arithmetic: GPT-3 could perform multi-digit addition and subtraction with reasonable accuracy, despite never being trained on a math curriculum
• Code generation: It could write functional code snippets in Python and JavaScript from natural language descriptions
• Translation: Few-shot GPT-3 matched or exceeded supervised baselines on some language pairs, particularly for translating into English
• News article generation: Given headlines, it produced articles that human evaluators struggled to distinguish from real news

The arithmetic capability deserves special attention because it cannot be explained by memorization. If GPT-3 had simply memorized a table of addition facts, it would get familiar problems right and novel problems wrong. Instead, performance degraded gradually as problem complexity increased, the hallmark of an imperfect learned procedure rather than a lookup table.

The arithmetic results are particularly interesting because they suggest the model learned algorithmic procedures rather than memorizing specific calculations. Performance degrades gracefully with more digits, the pattern you'd expect from imperfect procedure learning rather than lookup table failure. When a lookup table is exhausted, performance drops to zero. When a procedure is imperfect, performance drops gradually as the problem complexity exceeds the model's ability to apply the procedure reliably. GPT-3 shows the latter pattern, implying that whatever computation it performs for arithmetic has the character of a learned algorithm.

GPT-3 struggled on tasks requiring precise, structured outputs or multi-step reasoning:

• Reading comprehension with multiple choice: On RACE-h, GPT-3 achieved only 46.8% compared to 90% for fine-tuned models. The gap here is enormous and instructive. RACE-h consists of multiple-choice questions from Chinese high school reading comprehension tests. While GPT-3 can understand the English text, the task requires selecting from four options where three are plausible distractors. The model often fails to eliminate plausible-but-wrong options with the precision needed for reliable performance.
• Natural language inference: Performance on ANLI (adversarial NLI) remained below random chance for some splits. ANLI was specifically designed to be adversarially hard for models that rely on surface-level patterns. GPT-3's failure here reveals that its "reasoning" is often shallow, relying on word overlap and semantic similarity rather than valid logical inference.
• Math word problems: Multi-step reasoning with intermediate calculations proved challenging. Problems requiring the model to maintain intermediate results across multiple calculation steps fell apart as the number of steps increased, because the model had no mechanism for reliable working memory beyond what fit in its context.
• Fact verification: Distinguishing true from false claims required external knowledge verification the model could not perform.

These weaknesses pointed to fundamental limitations: GPT-3 excels at pattern completion and surface-level understanding but struggles with tasks requiring careful logical reasoning or precise knowledge retrieval. The model can approximate reasoning in familiar domains where training data provides dense coverage, but it lacks the systematic, step-by-step reasoning that humans use for structured problems.