GPT-3 › GPT-3 and the Problem of Reasoning

One of the clearest limitations revealed by GPT-3's evaluation is the gap between language understanding and systematic reasoning. This distinction is subtle but important, and GPT-3 helped clarify it by providing the first model capable enough to make the distinction crisp.

Language understanding refers to the ability to process and represent the meaning of text: parsing syntactic structure, resolving coreference, understanding word sense, and building representations of discourse. GPT-3 is excellent at this. It can read a complex passage and correctly answer questions about what happened, who did what to whom, and what the author intended. These tasks require understanding, but they can often be answered by pattern matching against familiar textual patterns.

Systematic reasoning refers to the ability to apply logical operations in a step-by-step, reliable way to derive conclusions from premises. This includes deductive reasoning (if P implies Q and P is true, then Q is true), abductive reasoning (finding the most likely explanation for observed evidence), and mathematical reasoning (applying arithmetic and algebraic operations in the correct order). GPT-3 struggles with these tasks not because it cannot understand the problem statement but because applying multi-step logical rules reliably requires a kind of precise, sequential computation that natural language training does not directly optimize for.

The contrast becomes particularly clear on syllogism tasks. Given "All mammals are warm-blooded. Dolphins are mammals. Are dolphins warm-blooded?", GPT-3 consistently answers correctly. The pattern is familiar: the training data is filled with examples of this kind of reasoning. But given "All glorks are flumps. Wumbles are glorks. Are wumbles flumps?", performance drops significantly. The logical structure is identical, but the words are novel, and the model cannot rely on stored associations about glorks, flumps, and wumbles. It must reason from pure logical structure, and here it begins to fail.

This observation motivated chain-of-thought prompting, developed shortly after GPT-3. The key insight is that if you show the model examples of step-by-step reasoning in your few-shot demonstrations, it generates step-by-step reasoning for new problems, and this substantially improves performance on multi-step tasks. The reasoning steps serve as an explicit working memory, externalizing the intermediate computation that the model cannot reliably maintain internally. Rather than trying to "think" through multiple steps in a single forward pass, the model generates text that represents each step, using the generated text as context for the next step.

Chain-of-thought prompting does not fully solve the reasoning problem, but it illustrates an important principle: careful prompt design can use what the model is already good at (generating coherent text) to address what it is bad at (maintaining multi-step computation internally). Many of the most effective prompt engineering techniques follow this pattern: they find ways to reformulate hard tasks as easier language generation tasks, exploiting the model's strengths to work around its limitations.