HellaSwag › What HellaSwag Reveals About Language Model Cognition

The trajectory of model performance on HellaSwag offers a window into how language models acquire and deploy knowledge about the physical world. Tracing this trajectory carefully reveals which models succeeded and why success required the specific combination of scale and training data that large language models brought to bear.

Early BERT-scale models approached HellaSwag with roughly 110 million to 340 million parameters trained on BookCorpus and Wikipedia. These pretraining sources contain rich vocabulary, grammatical diversity, and factual content, but they are thin on the procedural, step-by-step descriptions of physical activities that HellaSwag requires. A model that has primarily read encyclopedia articles and novels has seen relatively few descriptions of "now tighten the bolt until snug" or "fold the dough over and press firmly." Without dense exposure to procedural text, the model lacks the prior knowledge to evaluate which physical continuation is most natural.

The jump in performance at GPT-3 scale (175 billion parameters, trained on hundreds of billions of tokens from Common Crawl, books, and Wikipedia) coincides with massive expansion of the training corpus to include instructional websites, recipe databases, home improvement forums, and fitness tutorials. These sources are dense with exactly the kind of physical procedural knowledge that HellaSwag tests. A model that has processed millions of cooking recipes has strong priors about the sequence of steps in food preparation. A model that has processed thousands of home improvement guides knows which tools are used for which materials. The improvement in HellaSwag accuracy at scale is more than a matter of having more parameters to represent the same information; it reflects the acquisition of qualitatively different types of knowledge from qualitatively different text sources.

This observation has implications beyond HellaSwag itself. It suggests that the commonsense reasoning capabilities of large language models are not emergent from scale alone, but from scale in combination with the right type of pretraining data. Models trained exclusively on formal text, academic papers, or a narrow domain would likely fail to acquire the physical commonsense encoded in web-scale corpora, even at large parameter counts.

Probing studies have attempted to distinguish between memorized procedural patterns and physical reasoning in large language models, with mixed results that resist clean interpretation.

One line of evidence suggests pattern memorization. When researchers test models on HellaSwag variants that involve perturbing familiar activities in unfamiliar ways, such as replacing common tools with unusual substitutes that would work just as well, model accuracy drops significantly. This suggests that models have learned "activity scripts," stereotyped sequences of actions associated with particular activities, rather than general physical principles that would apply to novel configurations.

Another line of evidence suggests generalization. Models can correctly answer questions about physical scenarios that are unlikely to have appeared verbatim in training data, such as novel combinations of household objects or unusual physical situations. If models were purely relying on memorized scripts, this generalization would be impossible. The ability to reason about these novel cases implies some degree of general physical knowledge representation, even if that representation is not as reliable or flexible as human physical reasoning.

The most likely explanation is that large language models operate with a mixture of both mechanisms: they have internalized both specific procedural scripts (memorized from training data) and more general heuristics about physical plausibility (learned from statistical patterns across many activities). HellaSwag, by design, tests cases where the correct answer aligns with the most common procedural script for the described activity. This means it rewards both memorization of specific scripts and general physical reasoning, which makes it difficult to disentangle the two from accuracy alone.

HellaSwag belongs to a family of commonsense reasoning benchmarks that probe different aspects of physical and social world knowledge. Understanding how it relates to these neighbors clarifies what it uniquely measures.

The Winograd Schema Challenge and its scaled successor WinoGrande test pronoun disambiguation, requiring models to understand who is the subject of an action based on contextual physical or social constraints. While related to HellaSwag, these tasks require resolving reference rather than predicting continuations, and they draw on social and causal inference rather than purely physical knowledge.

The PIQA benchmark (Physical Intuition Question Answering) tests physical intuition through goal-oriented questions: given a goal and two candidate methods, which method achieves the goal? Like HellaSwag, PIQA requires understanding physical affordances and constraints. Unlike HellaSwag, it focuses on goal achievement rather than activity continuation, and its examples are not grounded in video footage, making the "correct" answer sometimes more subjective.

SWAG, HellaSwag's predecessor, used a similar activity completion format but without adversarial filtering. This makes SWAG much easier for language models trained on similar activity descriptions. The comparison between SWAG and HellaSwag performance for any given model quantifies how much adversarial filtering raises the bar: models that achieve 80% on SWAG may achieve only 35% on HellaSwag, illustrating the gap between artifact-exploiting performance and commonsense reasoning.

Understanding this family of benchmarks together provides a more complete picture of model commonsense capabilities than any single benchmark can offer alone. A model that excels at HellaSwag but fails at Winograd schemas has strong physical activity knowledge but weak pronoun disambiguation. A model that succeeds at both has broader commonsense coverage. The future of commonsense evaluation likely involves composite assessments across multiple specialized benchmarks rather than reliance on any single standard.