HellaSwag: Adversarial Filtering for Commonsense Reasoning

Common sense reasoning remains one of the most elusive capabilities in artificial intelligence. While language models can generate fluent prose, translate between languages, and memorize large repositories of factual knowledge, they often stumble on the intuitive physical reasoning that humans acquire effortlessly through lived experience. Consider a few simple scenarios: understanding that a thrown ball will arc downward under gravity, recognizing that a person holding a wet umbrella has likely been standing in rain, or knowing that opening a door requires first grasping the handle rather than pushing on the hinge side. These inferences, which humans make instantaneously and unconsciously, require a model to simulate physical causality, track object states through time, and reason about goal-directed human behavior in the world.

The challenge is not that these facts are obscure. Any person who has grown up interacting with physical objects learns these relationships through thousands of repeated experiences. The challenge is that language models do not have bodies, do not manipulate objects, and do not observe the continuous physical world. They learn from text alone, which captures human descriptions of the world but not the world itself. This gap between linguistic competence and physical grounding creates a basic tension in evaluating whether language models understand everyday activities or merely reproduce the surface patterns of how such activities are described in writing.

The HellaSwag benchmark emerged from a growing recognition within the natural language processing community that existing natural language inference datasets had become saturated by surface-level patterns and annotation artifacts. Datasets such as SNLI (Stanford Natural Language Inference) and MultiNLI initially provided rigorous tests of textual entailment, but researchers soon discovered that models could achieve surprisingly high accuracy by exploiting subtle patterns introduced during dataset construction. Negative examples in these datasets often contained specific linguistic markers such as negation words or particular verb tenses that allowed models to classify correctly without truly understanding the semantic relationships between premise and hypothesis. The models were cheating, in effect, by recognizing symptoms of incorrect answers rather than the substance of the reasoning itself. This gap between benchmark performance and understanding motivated the creation of a more reliable evaluation framework that could resist such superficial shortcuts.

The problem was not that dataset creators were careless. Human annotators writing incorrect examples naturally introduce subtle differences from the correct examples, even when trying to be consistent. The very act of constructing a wrong answer leaves traces in the writing: wrong answers tend to be slightly less specific, use slightly different vocabulary patterns, or employ subtly different grammatical constructions. These differences are invisible to human readers but detectable by statistical models trained to identify them. Adversarial filtering directly targets this problem.

HellaSwag, which stands for Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations, introduced a novel approach to dataset construction called adversarial filtering. This technique uses machine learning models themselves to iteratively identify and remove examples that rely on spurious statistical cues. The result is a multiple-choice completion task where models must select the most plausible continuation of an everyday activity described in a short narrative context. Even as large language models surpassed human performance on earlier benchmarks like SNLI and MultiNLI through progressive scaling and architectural improvements, HellaSwag remained stubbornly difficult. Early transformer models, despite their impressive capabilities on other tasks, scored barely above random chance on HellaSwag, even though the task appears deceptively simple to human readers who find it almost trivially easy.

The benchmark draws its examples from ActivityNet captions describing mundane human activities: cooking meals, exercising, crafting objects, and performing household chores. Each example presents a short context describing the beginning of an activity, followed by four possible endings. Only one ending represents the actual continuation captured from the video; the other three are adversarially generated distractors designed to be grammatically plausible but physically impossible or contextually inconsistent. This adversarial construction process ensures that success requires understanding causal relationships, physical constraints, and temporal coherence rather than merely matching n-gram patterns or measuring lexical overlap between context and ending.

What makes HellaSwag especially interesting from a benchmark design perspective is that it uses two complementary sources of difficulty. The first is the intrinsic difficulty of the commonsense reasoning task: human activities involve complex physical interactions that are difficult to predict without world knowledge. The second is the adversarially constructed difficulty imposed by the filtering process: even if the task were simpler, the filtering ensures that easy examples (those solvable by surface heuristics) have been removed. Together these two sources of difficulty create a benchmark where models must both understand the physical world and resist the temptation to use statistical shortcuts. This combination proved more reliable than either source of difficulty alone would have been, keeping the benchmark challenging for several years after its creation.

Understanding HellaSwag also illuminates a broader question that the NLP community grappled with in the years following its publication: what does it mean for a language model to "understand" text, as opposed to merely processing it statistically? The gap between early model performance and human performance on HellaSwag was concrete evidence that something qualitatively different separated human cognition from statistical language modeling in 2019. Watching that gap close as models scaled up forced the community to refine its theories about what kind of understanding the task requires for the task and whether the models that eventually closed the gap had achieved understanding or merely a more powerful simulation of it.

To understand what makes HellaSwag challenging, we must examine the precise structure of its task design. Unlike reading comprehension benchmarks where the answer appears explicitly in the text, or natural language inference tasks that rely primarily on textual entailment patterns, HellaSwag requires models to simulate the physical world implied by the text. The model must construct a mental model of the scene, track the objects present and their configurations, and predict which physical transformations are physically possible given the described state of affairs.

This demands a specific form of causal reasoning that goes beyond pattern completion. When a human reads that someone is holding a knife and an orange, they immediately activate a rich network of knowledge about kitchen activities, cutting procedures, the physical properties of citrus fruit, and the goals that motivate food preparation. They understand that oranges can be sliced but not drilled, that knives cut solid objects but do not stir liquids, and that the natural progression of food preparation follows recognizable procedural patterns. Encoding this knowledge from text alone requires either explicit injection of physical world knowledge or implicit acquisition through massive exposure to descriptions of physical activities.

Each HellaSwag instance consists of three core components. First, there is a context: typically four sentences describing the initial steps of an activity, drawn from video captions in the ActivityNet dataset. These sentences establish the physical setting, the objects present, and the initial actions performed by the subject. Second, there are four candidate endings: one gold (correct) ending from the original video caption, and three negative (incorrect) endings generated by language models and filtered adversarially to maximize difficulty. Third, there is the evaluation objective: select the ending that best continues the activity in a physically and logically coherent manner, maintaining consistency with the objects, actions, and goals established in the context.

The adversarial generation process ensures that incorrect endings are "close misses" rather than obvious nonsense. A distractor might describe a plausible action that happens to violate physical constraints, such as pouring liquid into a sealed container, or contradict the established context, such as using tools that were never mentioned in the setup. These distractors maintain grammatical correctness and lexical coherence, making them difficult to filter based on surface linguistic features alone. A distractor that uses the same vocabulary as the context while describing a physically impossible action is far harder to reject through statistical pattern matching than a distractor that uses completely different vocabulary.

Consider what makes this difficult for language models. Traditional language modeling objectives optimize for predicting likely next tokens based on statistical patterns observed in training data. Models learn co-occurrence statistics between words and phrases, letting them to generate fluent text that resembles human writing. However, HellaSwag requires counterfactual reasoning: the model must distinguish likely word sequences from physically possible actions in the described state of the world. The model must ask: "Given that the knife is currently on the counter, can the next action involve cutting with that knife?" This distinction between statistical correlation and causal understanding lies at the heart of the benchmark's design philosophy.

HellaSwag's breakthrough contribution was the adversarial filtering algorithm used to construct the dataset. Prior commonsense reasoning datasets suffered from persistent annotation artifacts: subconscious patterns in how humans write distractors that models could eventually learn to exploit. For instance, negative examples might systematically use different verb tenses than positive examples, contain negation words like "not" or "never," or exhibit lower lexical overlap with the premise than positive examples do. Once researchers identified these biases, models could be trained specifically to detect them, achieving high accuracy on the benchmark without developing an understanding of the underlying reasoning task. This created a mirage of progress, where improving benchmark scores did not translate to improved real-world reasoning capabilities.

Adversarial filtering addresses this basic vulnerability by using a dynamic, iterative process where models themselves identify which examples are solvable through surface cues, allowing researchers to remove or modify those examples until only difficult instances remain. The process functions as a computational arms race, where each iteration attempts to eliminate the shortcuts discovered by the previous generation of models. Rather than asking "what patterns might models exploit?" from a human perspective, the process asks models directly to show what patterns they exploit, then removes those patterns.

The adversarial filtering process operates through several carefully designed stages that work together to produce a progressively more reliable dataset.

The first stage is candidate generation. The algorithm begins by generating candidate distractors for each context. Early versions used simple n-gram language models or retrieval from similar contexts to create initial negative examples. The key insight is that these initial distractors, while often obviously wrong to human readers, may contain subtle patterns that discriminate them from correct endings when viewed through the lens of statistical learning. What matters at this stage is quantity: generating a large pool of candidate distractors that will be progressively filtered.

The second stage is model training and discrimination. A discriminator model, typically a pretrained transformer like BERT, is trained to distinguish gold endings from the generated distractors. This model is a probe for dataset artifacts: if the discriminator achieves high accuracy after only brief training, it has likely discovered exploitable patterns in the distractors rather than learning the true underlying task. The discriminator's successes are diagnostic: they reveal exactly which examples contain detectable artifacts, pointing toward the specific features that must be eliminated or masked.

The third stage is distractor re-generation. For examples where the discriminator succeeds, new distractors are generated using more sophisticated models or constrained sampling techniques. The goal is to create adversarial negatives that fool the current discriminator while remaining clearly incorrect to human judges. This might involve so that distractors match the context in terms of verb tense, noun overlap, or syntactic structure. The constraints are informed directly by what the discriminator previously used to distinguish correct from incorrect examples.

The fourth stage is iterative refinement. Stages 2 and 3 repeat iteratively, with increasingly powerful discriminators and generators, until the dataset reaches a target difficulty level where even strong models struggle to distinguish gold from generated endings. The final dataset contains examples where state-of-the-art models at the time of creation performed only marginally better than random chance.

This process creates a productive tension between the generator and the discriminator, with the dataset benefiting from the failures of both systems. When the discriminator finds a shortcut, such as noticing that gold endings often contain words from the context arranged in a particular order, the generator must produce harder negatives that break that specific pattern. The resulting dataset is adversarially reliable against the class of models used in its construction, though not necessarily against all possible models or reasoning strategies. This conditionality on the discriminator pool becomes important when interpreting later results.

Adversarial filtering provides no formal guarantee that undiscovered shortcuts do not exist within the final dataset. The technique only ensures that specific model architectures, namely those used as discriminators during construction, cannot solve the task through the patterns those discriminators identified. This limitation leads to an important dynamic in benchmark evaluation: when new architectures emerge, such as the transition from LSTMs to transformers, or from BERT to GPT-3 scale models, they may suddenly "break" the adversarial constraints by discovering new types of reasoning or new artifacts not anticipated by the filtering process.

Think of it as a security analogy. Adversarial filtering is like patching known vulnerabilities in a system. Each filtering round closes one set of exploits, but unknown exploits may still exist. A more powerful attacker, analogous to a more capable language model, may find entirely different attack vectors that the original patches never addressed. The system is secure against known attacks, not against all possible attacks.

The effectiveness of adversarial filtering depends critically on the diversity of the discriminator pool. If all discriminators share similar inductive biases, they may collectively miss certain types of shortcuts that exploit those shared biases. HellaSwag addresses this concern by using an ensemble of different model architectures and training procedures during the filtering phase. This keeps the dataset resists a broader range of potential exploitation strategies. Nevertheless, all models in 2018-2019 shared one important bias: they had relatively limited contextual understanding compared to models trained at the scale that emerged in 2020 and beyond. This shared limitation meant that certain sophisticated reasoning patterns remained outside the reach of the discriminators. This leaves corresponding shortcuts undetected.

HellaSwag sources its contexts from ActivityNet, a large-scale video understanding dataset containing human-annotated descriptions of everyday activities. These captions provide naturally occurring descriptions of procedural knowledge: how to make a sandwich, change a tire, or perform a yoga pose. The video origin is important because it grounds the text in physical reality; the gold endings describe actions that occurred and were visually verified by human annotators who watched the video footage. This connection to physical reality ensures that the "correct" answer corresponds to observed events in the world that a human observer confirmed by watching them happen, rather than a matter of opinion or stylistic preference.

The ActivityNet dataset contains approximately 20,000 videos spanning 200 activity categories, with each video accompanied by dense temporal annotations describing what happens at different points in the activity. HellaSwag selects from this pool using criteria designed to ensure the completion task remains well-defined and challenging.

Contexts are selected to satisfy several strict criteria that together ensure the task remains well-defined and challenging:

• Temporal coherence: The context must describe a continuous activity without jumps in time or location. The narrative should flow logically from one action to the next, establishing a clear physical trajectory that the ending must continue.
• Physical grounding: The activity must involve observable physical actions rather than abstract reasoning or emotional states. This ensures that the task tests physical commonsense rather than theory of mind or emotional intelligence, keeping the evaluation focused on a specific and tractable form of reasoning.
• Completion ambiguity: The context must be sufficiently open-ended that multiple continuations appear plausible to a reader who has not seen the video, but sufficiently constrained that physical constraints eliminate most physically impossible options. If only one continuation is remotely plausible, the task becomes trivial. If too many are plausible, the task becomes ambiguous.

The average context length of 35-40 words provides enough detail to establish the physical situation without revealing the ending. This length was determined empirically through pilot studies: shorter contexts allowed too many plausible continuations, making the task ambiguous even for humans, while longer contexts made the task trivial by revealing the outcome or the specific objects involved in the final steps.

This balance between ambiguity and constraint is a subtle design achievement. Consider what happens at the extremes. A one-sentence context such as "A person holds a hammer" is too underspecified: hundreds of activities involve hammers, and the correct continuation could be any of them. A ten-sentence context that describes an entire nailing procedure in complete detail is so specific that only one continuation is plausible for anyone who has ever used a hammer. The 35-40 word sweet spot preserves just enough ambiguity that naive pattern-matching fails while giving just enough specificity that the correct continuation is uniquely determined by physical reasoning. Finding this balance required iterating over many potential context lengths with human judges, adjusting until inter-annotator agreement on the correct ending was consistently high while model performance remained near chance.

The adversarial generation employs three complementary strategies to create challenging distractors that resist simple statistical detection. Each strategy targets a different dimension of physical reasoning. This keeps the dataset tests multiple types of commonsense knowledge.

Generative model sampling uses language models to produce completions conditioned on the context. Early HellaSwag construction used GPT-1 and GPT-2 scale models as generators. These models produce plausible-sounding narratives but frequently hallucinate objects or actions inconsistent with the physical setup described in the context. A generated distractor might use a tool that was not mentioned, reference a material that has different properties than implied, or describe an action in the wrong order. After adversarial filtering, the distractors that survive are those that are plausible enough to fool a discriminator while still being incorrect.

Counterfactual editing involves modifying gold endings by changing key actions or objects. Substituting "pour" for "stir," or replacing "bowl" with "bag," creates physically impossible but grammatically coherent scenarios. These edits test whether models understand the functional affordances of objects: what a knife can and cannot do, what a bag can and cannot hold, which actions are reversible and which are not. Counterfactual editing is particularly effective at targeting models that reason about physical properties, because the edited distractors preserve every surface feature of the correct ending except the specific word that encodes the physical constraint.

Retrieval from similar contexts takes endings from different but semantically similar contexts and presents them as options for the target context. An ending about "baking the mixture" might be retrieved for a context that only mentions cold preparation, or an ending about "inserting screws" might be retrieved for a context involving adhesive assembly. These retrieved distractors are physically plausible in the abstract but violate the specific constraints established by the context. They test whether models track the state of the physical world described in the context rather than just recognizing that the ending describes a physically possible action in general.

Each generated distractor undergoes automatic filtering to ensure grammatical correctness and surface plausibility before entering the adversarial filtering loop. This pre-filtering ensures that the adversarial process focuses on semantic and physical coherence rather than basic linguistic validity. A distractor that is grammatically incoherent is trivially rejected by any model; the interesting distractors are those that are grammatically perfect but physically wrong.

HellaSwag employs a straightforward multiple-choice accuracy metric, but the interpretation of this metric requires understanding the baseline performance levels and the significant gap between random guessing and human performance. This gap is a measure of the task's inherent difficulty and the effectiveness of the adversarial filtering process. Unlike benchmarks where near-ceiling human performance suggests the task is too easy, HellaSwag's design deliberately targets the large initial gap between model and human performance as evidence that the adversarial approach succeeded.

The choice of multiple-choice format over open-ended generation is deliberate and consequential. Open-ended generation would require a reference-based metric or human evaluation to assess whether the model's continuation is physically coherent, and both options introduce noise: reference-based metrics penalize valid continuations that differ from the reference, while human evaluation is expensive and slow. Multiple-choice format allows fully automatic evaluation: accuracy is deterministic and requires no human judgment once the dataset is constructed. This design choice trades off some generative capability measurement for scalability and reproducibility, a trade-off that has defined the mainstream of NLP evaluation since the rise of datasets like SNLI. HellaSwag inherits this trade-off deliberately, prioritizing the ability to run large-scale systematic evaluations over the richer information that free-form generation assessment would provide.

Formally, given a context $c$ and a set of four candidate endings $\{e_1, e_2, e_3, e_4\}$ where $e^*$ denotes the gold ending, a model assigns a score $s(c, e_i)$ to each candidate. The predicted ending is:

where the score function $s$ depends on the model architecture and evaluation strategy.

The overall accuracy on the test set of $N$ examples is:

where:

• $N$: the total number of evaluation examples (10,042 in the validation set)
• $\hat{e}_j$: the model's predicted ending for example $j$
• $e^*_j$: the gold ending for example $j$
• $\mathbf{1}[\cdot]$: the indicator function, equal to 1 when the condition is true and 0 otherwise

Different model architectures compute the score $s(c, e_i)$ differently, which reflects their underlying training objectives and inductive biases:

• Masked language models (BERT): Concatenate context and ending with a [SEP] token, use the [CLS] representation as input to a linear classifier, or compute pseudo-log-likelihood by masking and scoring each token in the ending. This approach treats the task as discriminative classification.
• Autoregressive models (GPT): Compute the conditional probability $P(e_i \mid c)$ by concatenating context and ending, then summing log probabilities of the ending tokens given the context. Let the context tokenize to positions $1, \ldots, L$ and the ending to positions $L+1, \ldots, L+M$. The score is:

where $x_t$ is the token at position $t$. This approach treats the task as conditional language modeling rather than classification.

• Encoder-decoder models: Encode the context and decode the ending, scoring based on reconstruction probability or sequence-to-sequence alignment loss. The ending that the model assigns the highest generation probability to is selected as the predicted continuation.

An important subtlety arises with length normalization. Longer endings accumulate more log-probability terms and therefore tend to receive lower raw scores simply because they have more tokens to predict. To control for this, practitioners often normalize by ending length:

where:

• $M_i$: the number of tokens in ending $e_i$
• $L$: the number of tokens in the context $c$
• $x_t$: the token at position $t$ in the concatenated sequence

Length normalization prevents the model from systematically preferring shorter endings regardless of their content, which would otherwise create an exploitable artifact in the evaluation itself.

Random chance accuracy on HellaSwag is 25%. This reflects the four-way multiple choice design. Early results established important reference points that demonstrated the effectiveness of adversarial filtering:

• Bag-of-words baseline: 31% accuracy, showing that lexical overlap between context and ending provides some signal but falls far short of solving the task.
• BERT-base (zero-shot): 33% accuracy, showing slight improvement over lexical methods but still approaching random chance despite BERT's strong general language understanding capabilities.
• GPT-2 (fine-tuned): approximately 48% accuracy, showing that larger autoregressive models capture more physical regularities but still fall well below human performance.
• Human performance: 95% accuracy, near-ceiling performance showing that the task is well-defined and solvable for humans with ordinary commonsense knowledge.

The massive gap between early model performance (33%) and human performance (95%), spanning 62 percentage points, indicated that the adversarial filtering had successfully removed superficial cues that might have allowed statistical models to shortcut the reasoning process. This gap became the target for subsequent research and model development, motivating work on better commonsense knowledge integration, larger pretraining corpora, and improved architectures.

The visualization reveals a striking pattern. Model performance stagnated near 30-33% for the first generation of BERT-scale models, improved modestly to around 48% for fine-tuned GPT-2, and then jumped dramatically with GPT-3 scale models to nearly 80%. This non-linear progression suggests that a threshold of parameter count, training data volume, or representational capacity must be crossed before models can effectively simulate physical world dynamics from text alone. Below that threshold, models are essentially pattern-matching with insufficient capacity to capture the relevant physical regularities. Above it, models have seen enough descriptions of physical activities that they can interpolate successfully even for novel combinations.

Walking through a concrete HellaSwag example makes the reasoning requirements tangible and reveals why certain distractors are particularly challenging. Let us examine an example that illustrates the different types of physical commonsense the task demands.

Context:

"A person is seen standing in a kitchen holding a knife and an orange. They begin by cutting the orange in half. They then take one half and begin slicing it into smaller pieces. They place the sliced orange pieces into a bowl."

Candidate Endings:

1. "They then take the other half and continue slicing it into pieces, adding them to the bowl."
2. "They then pour the orange juice into a glass and drink it."
3. "They then put the knife in the refrigerator to keep it fresh."
4. "They then use the knife to stir the orange slices in the bowl."

Analysis:

Option 1 is the gold ending. It continues the established activity logically, using the "other half" that was implicitly left over when the first half was sliced. It maintains the physical setup (knife, bowl), extends the pattern established by the previous actions (slicing pieces into the bowl), and completes the obvious goal of processing the entire orange. Notice how this ending requires understanding that "the other half" refers to the half not yet processed, a form of object tracking that requires understanding the physical state left by the preceding actions.

Option 2 describes a plausible outcome in some contexts, since oranges can be juiced. However, it introduces a glass that was never mentioned in the context, and it skips the necessary squeezing or juicing step that would be required to produce liquid from sliced pieces. More subtly, the context describes a slicing procedure, and the transition to pouring liquid juice is physically inconsistent with having produced solid sliced pieces rather than squeezed juice. This distractor tests whether models track the physical state of the orange (now sliced into solid pieces) rather than just recognizing that orange juice is a common outcome of working with oranges.

Option 3 violates physical commonsense entirely: knives are not refrigerated to keep them fresh, and refrigeration is not a step in orange preparation. This distractor maintains grammatical correctness and uses objects from the context (the knife), but describes an action that contradicts basic knowledge of kitchen practices and the preservation of metal tools. It is the easiest distractor to reject for a model with any physical world knowledge, but it is a useful sanity check.

Option 4 tests knowledge of tool affordances: knives are not used for stirring, and attempting to stir with a knife blade would be awkward and potentially dangerous. This distractor maintains both objects from the context (knife, bowl) and describes an interaction between them, which makes it superficially plausible to a model that tracks object co-occurrence without understanding functional properties. A model that merely checks "does this ending mention objects from the context?" would be equally drawn to Options 1 and 4, requiring physical knowledge about tool use to distinguish them.

The key insight from this analysis is that each distractor tests a different dimension of physical commonsense. Option 2 tests object state tracking, Option 3 tests basic physical plausibility, and Option 4 tests tool affordances. The adversarial filtering process is designed to ensure that distractors of this caliber, which require physical reasoning to reject, dominate the final dataset rather than distractors that can be rejected through simple lexical or syntactic cues.

This layering of different reasoning requirements across distractors is not accidental. The HellaSwag construction process generates multiple candidate distractors per example and selects the hardest surviving ones after filtering. The result is that each example often contains at least one distractor that is close to the gold ending in every measurable surface feature except physical plausibility, forcing models to engage with the substance of the physical scenario rather than its description.

Let us implement a HellaSwag evaluation pipeline to see how models are scored in practice. We will load the dataset, prepare examples for a language model, and compute accuracy using the conditional log-probability scoring approach described above.

First, we install and import the necessary libraries:

The results reveal the characteristic HellaSwag pattern: models exceed random chance by a modest margin. This shows that statistical regularities in language do encode some physical world knowledge, but a large gap to human performance remains. The zero-shot approach (no fine-tuning on HellaSwag) is intentional: we want to measure what physical knowledge the model has acquired through general language modeling, not how well it can learn the statistical patterns of the benchmark itself through task-specific fine-tuning.

To understand how confidently the model distinguishes correct from incorrect endings, we can examine the distribution of scores for correct versus incorrect options:

The score distribution plot illustrates a basic limitation of small language models on HellaSwag: the distributions for correct and incorrect endings overlap substantially. A model with perfect physical reasoning ability would assign clearly higher scores to correct endings, creating well-separated distributions. The observed overlap means that for many examples, the model essentially guesses: the correct ending happens to receive the highest score by a narrow margin rather than by confident discrimination. Larger, more capable models show greater separation between these distributions, showing more confident and accurate physical plausibility judgments.

Understanding the configurable aspects of HellaSwag evaluation helps interpret results across different settings and ensures comparisons are meaningful:

• model_name: The pretrained model identifier (for example, "gpt2", "gpt2-xl", "meta-llama/Llama-2-7b"). Larger models generally perform better but require more computation and memory.
• max_length: Maximum token length (commonly 512 or 1024) for context and ending concatenation. Longer sequences are truncated. Truncation can affect results if contexts exceed the limit, though this is rare given the typical context length.
• length_normalization: Whether to normalize log probabilities by ending length. This is recommended to avoid systematic bias toward shorter endings, but some published results omit it.
• max_examples: Number of examples to run. The full validation set contains 10,042 examples. Smaller samples introduce sampling variance, so results on fewer than 500 examples should be interpreted with caution.
• device: Computation device ("cuda" or "cpu"). GPU acceleration significantly speeds up inference for neural language models.
• random_baseline: Expected accuracy for uniform random guessing is 0.25, since there are four options by construction.
• human_performance: Human annotators achieve approximately 95% accuracy, serving as the practical upper bound for the benchmark.

HellaSwag was designed specifically to avoid the fate of earlier NLI datasets, yet it too has approached saturation as model capabilities have advanced. GPT-3 achieved 78.9% accuracy in the original few-shot setting, PaLM reached 85.4%, and subsequent models including GPT-4, Claude 3, and Llama 3 have pushed scores into the 93 to 95% range, essentially matching or exceeding average human performance. This rapid progression from "unsolved" to "saturated" illustrates a basic challenge in AI evaluation: the iterative nature of adversarial filtering creates a snapshot of difficulty based on contemporary model limitations, but this snapshot becomes obsolete as architectures improve and scale increases.

The progression is instructive. HellaSwag was released in 2019 when BERT-scale models could barely exceed random chance on it. Within five years, models trained with hundreds of billions of parameters had closed the entire gap to human performance. That shift raises a more interesting question than whether models "passed" the benchmark: what capability change made the task suddenly tractable, and does that capability reflect physical understanding or a more powerful form of pattern matching?

Several competing explanations exist for why large language models eventually mastered HellaSwag, and the evidence supports a combination of all of them rather than a single definitive answer.

The first explanation is memorization of training data. ActivityNet videos have associated web pages, YouTube descriptions, and commentary. A model trained on internet-scale data may have seen descriptions of specific activities that appear in HellaSwag or paraphrases of them. If a model has processed 500 billion tokens of text, the odds that it encountered descriptions of "slicing an orange into pieces and placing them in a bowl" approach certainty. This is not the same as physical reasoning; it is pattern retrieval from a very large associative memory.

The second explanation is statistical generalization of physical patterns. Even without memorizing specific HellaSwag examples, a model trained on enough procedural text, cooking blogs, instructional manuals, and activity descriptions may develop strong priors about how physical activities typically unfold. These priors would not constitute a formal model of physical reality, but they would be sufficient to select the most statistically typical continuation for everyday activities, which is precisely what HellaSwag measures.

The third explanation is emergent physical reasoning. As models grow larger, they may develop internal representations that approximate reasoning about physical states, object affordances, and causal sequences. Evidence for this comes from the fact that large models generalize to novel physical scenarios not covered by their training data and can answer questions about hypothetical situations involving objects and actions they have never seen described together.

These explanations are not mutually exclusive, and current interpretability methods cannot fully disentangle them. The practical implication is that high HellaSwag accuracy does not cleanly certify that a model has achieved human-like physical commonsense. It certifies that the model has acquired, through some combination of these mechanisms, the ability to perform at human level on this particular benchmark.

Benchmark saturation manifests in several observable ways that researchers should monitor:

Accuracy ceiling: When top models cluster within a few percentage points of human performance and of each other, diminishing returns suggest the ceiling has been reached. On HellaSwag, the gap between GPT-4 (94.2%) and human performance (95%) is narrow enough that error analysis becomes more informative than headline scores. When improvements become incremental, the benchmark no longer discriminates effectively between model capabilities.

Error correlation with annotation ambiguity: Examining remaining failures often reveals annotation errors or ambiguous examples rather than clear model failures. In saturated regimes, model mistakes frequently correspond to cases where human annotators disagree, or where the "correct" answer is debatable upon careful inspection. When errors correlate with human disagreement rather than systematic model weaknesses, the benchmark has likely been saturated.

Transfer to harder variants: Researchers have proposed harder variants of HellaSwag by applying adversarial filtering with stronger modern models. If performance drops significantly on these filtered subsets, the original benchmark has likely been saturated by pattern matching rather than understanding. These harder variants serve as stress tests for whether models have truly mastered the underlying reasoning or merely memorized solution patterns from the original distribution.

The saturation pattern changes the interpretation for how researchers should interpret and report HellaSwag results. A new model achieving 94% accuracy in 2025 communicates much less information than a model achieving 94% accuracy in 2019, because the former has joined a crowded cluster near the ceiling while the latter would have been a large breakthrough. Benchmark scores should always be interpreted relative to the current state of the art and the historical trajectory, not as absolute measures of capability.

When a benchmark like HellaSwag saturates, we must ask what has been measured and what the saturation reveals about model capabilities. The adversarial filtering process ensures that models cannot rely on the specific artifacts known at creation time, but it cannot prevent models from discovering new, more sophisticated shortcuts or from acquiring the underlying reasoning capability through training on large corpora that implicitly contain similar physical reasoning patterns.

The evidence suggests a mix of both mechanisms is at work. Probing studies show that models make errors on HellaSwag variants that require detailed physical simulation, such as predicting object trajectories, understanding complex containment relations, or reasoning about quantities and materials. This suggests that some high performance comes from memorizing procedural patterns from training data rather than building a general physical simulator. However, the consistent improvement across diverse model architectures suggests that scaling and improved training have also conferred commonsense reasoning capabilities, or at least more reliable representations of physical causality that generalize beyond memorized examples.

Saturation also raises necessary questions about data contamination. If models are trained on internet-scale data that includes HellaSwag examples or paraphrases, high accuracy may reflect memorization rather than generalization. The HellaSwag authors released the dataset publicly, which means every model trained on web-scraped data after 2019 may have encountered the exact examples. Some organizations have begun releasing benchmark data only under restricted access specifically to avoid this contamination problem, though this creates its own tensions with reproducibility and open science norms.

The trajectory of model performance on HellaSwag offers a window into how language models acquire and deploy knowledge about the physical world. Tracing this trajectory carefully reveals which models succeeded and why success required the specific combination of scale and training data that large language models brought to bear.

Early BERT-scale models approached HellaSwag with roughly 110 million to 340 million parameters trained on BookCorpus and Wikipedia. These pretraining sources contain rich vocabulary, grammatical diversity, and factual content, but they are thin on the procedural, step-by-step descriptions of physical activities that HellaSwag requires. A model that has primarily read encyclopedia articles and novels has seen relatively few descriptions of "now tighten the bolt until snug" or "fold the dough over and press firmly." Without dense exposure to procedural text, the model lacks the prior knowledge to evaluate which physical continuation is most natural.

The jump in performance at GPT-3 scale (175 billion parameters, trained on hundreds of billions of tokens from Common Crawl, books, and Wikipedia) coincides with massive expansion of the training corpus to include instructional websites, recipe databases, home improvement forums, and fitness tutorials. These sources are dense with exactly the kind of physical procedural knowledge that HellaSwag tests. A model that has processed millions of cooking recipes has strong priors about the sequence of steps in food preparation. A model that has processed thousands of home improvement guides knows which tools are used for which materials. The improvement in HellaSwag accuracy at scale is more than a matter of having more parameters to represent the same information; it reflects the acquisition of qualitatively different types of knowledge from qualitatively different text sources.

This observation has implications beyond HellaSwag itself. It suggests that the commonsense reasoning capabilities of large language models are not emergent from scale alone, but from scale in combination with the right type of pretraining data. Models trained exclusively on formal text, academic papers, or a narrow domain would likely fail to acquire the physical commonsense encoded in web-scale corpora, even at large parameter counts.

Probing studies have attempted to distinguish between memorized procedural patterns and physical reasoning in large language models, with mixed results that resist clean interpretation.

One line of evidence suggests pattern memorization. When researchers test models on HellaSwag variants that involve perturbing familiar activities in unfamiliar ways, such as replacing common tools with unusual substitutes that would work just as well, model accuracy drops significantly. This suggests that models have learned "activity scripts," stereotyped sequences of actions associated with particular activities, rather than general physical principles that would apply to novel configurations.

Another line of evidence suggests generalization. Models can correctly answer questions about physical scenarios that are unlikely to have appeared verbatim in training data, such as novel combinations of household objects or unusual physical situations. If models were purely relying on memorized scripts, this generalization would be impossible. The ability to reason about these novel cases implies some degree of general physical knowledge representation, even if that representation is not as reliable or flexible as human physical reasoning.

The most likely explanation is that large language models operate with a mixture of both mechanisms: they have internalized both specific procedural scripts (memorized from training data) and more general heuristics about physical plausibility (learned from statistical patterns across many activities). HellaSwag, by design, tests cases where the correct answer aligns with the most common procedural script for the described activity. This means it rewards both memorization of specific scripts and general physical reasoning, which makes it difficult to disentangle the two from accuracy alone.

HellaSwag belongs to a family of commonsense reasoning benchmarks that probe different aspects of physical and social world knowledge. Understanding how it relates to these neighbors clarifies what it uniquely measures.

The Winograd Schema Challenge and its scaled successor WinoGrande test pronoun disambiguation, requiring models to understand who is the subject of an action based on contextual physical or social constraints. While related to HellaSwag, these tasks require resolving reference rather than predicting continuations, and they draw on social and causal inference rather than purely physical knowledge.

The PIQA benchmark (Physical Intuition Question Answering) tests physical intuition through goal-oriented questions: given a goal and two candidate methods, which method achieves the goal? Like HellaSwag, PIQA requires understanding physical affordances and constraints. Unlike HellaSwag, it focuses on goal achievement rather than activity continuation, and its examples are not grounded in video footage, making the "correct" answer sometimes more subjective.

SWAG, HellaSwag's predecessor, used a similar activity completion format but without adversarial filtering. This makes SWAG much easier for language models trained on similar activity descriptions. The comparison between SWAG and HellaSwag performance for any given model quantifies how much adversarial filtering raises the bar: models that achieve 80% on SWAG may achieve only 35% on HellaSwag, illustrating the gap between artifact-exploiting performance and commonsense reasoning.

Understanding this family of benchmarks together provides a more complete picture of model commonsense capabilities than any single benchmark can offer alone. A model that excels at HellaSwag but fails at Winograd schemas has strong physical activity knowledge but weak pronoun disambiguation. A model that succeeds at both has broader commonsense coverage. The future of commonsense evaluation likely involves composite assessments across multiple specialized benchmarks rather than reliance on any single standard.

While HellaSwag represented a significant advance over prior commonsense benchmarks, it carries important limitations that inform how we interpret its results and design future evaluations. Recognizing these limitations prevents overinterpretation of benchmark scores and guides the development of more complete evaluation protocols.

Narrow domain coverage: By sourcing exclusively from ActivityNet, HellaSwag focuses heavily on physical, procedural activities involving object manipulation and bodily movement. It does not test social reasoning, emotional intelligence, abstract causal reasoning, or the kind of contextual judgment required in professional or interpersonal settings. A model could excel at HellaSwag while failing to understand social conventions, emotional causality, or interpersonal dynamics. The benchmark captures only one slice of the broader commonsense knowledge domain, and that slice happens to be one that physical activities in video form represent well.

Cultural and linguistic bias: The dataset derives exclusively from English video captions describing primarily Western cultural practices. Commonsense reasoning in other cultures may involve different physical practices, tool uses, or social contexts not captured by these activity descriptions. The assumption that HellaSwag measures "universal" physical commonsense is parochial; it measures commonsense within a specific cultural and linguistic context. This limits the benchmark's utility for evaluating multilingual models and may introduce cultural biases about what constitutes "common" sense.

Static adversaries locked to 2018-2019 models: The adversarial filtering used models available at the time of construction, primarily BERT and early GPT models. Modern models with billions of parameters and internet-scale pretraining may solve HellaSwag through capabilities not anticipated by the filtering process, including memorization of specific video sources, more sophisticated n-gram statistics that capture physical regularities, or emergent reasoning abilities that arise only at large scale. The adversarial constraints that made the benchmark hard for 2018 models provide weaker guarantees against 2024 models.

Binary success metric obscures partial understanding: The multiple-choice format reduces fine-grained reasoning to binary correctness. A model might partially understand a scenario, recognizing that Option 3 (refrigerating the knife) is wrong, while failing to distinguish between two plausible continuations, Options 1 and 4. The accuracy metric loses this granularity: a model that always gets the wrong option wrong but cannot pick between two plausible options receives the same score as a model that fails randomly. Finer-grained metrics that reward partial credit or measure the ranking of all four options would provide more diagnostic information.

Distributional shift robustness not tested: HellaSwag tests models on examples drawn from the same distribution as the ActivityNet source, specifically everyday activities performed in typical settings. Real-world commonsense reasoning requires handling novel situations outside training distributions: unusual activity combinations, edge cases, adversarial perturbations, or activities in unusual contexts. The benchmark does not measure robustness to distribution shift or out-of-distribution generalization, which may be more relevant to real-world deployment.

Fixed dataset cannot adapt to improving models: Unlike human intelligence testing, which can introduce new question types when existing types become too easy, a static benchmark like HellaSwag becomes progressively less informative as models improve. Once the frontier model cluster reaches 94%, the benchmark can no longer rank models within that frontier. Continuous evaluation paradigms that automatically generate new examples or adaptively increase difficulty represent one approach to addressing this limitation, though they introduce their own challenges of reproducibility and consistency.

Despite these limitations, HellaSwag served its intended purpose admirably. It provided a difficult, artifact-resistant benchmark that drove years of research in commonsense reasoning and demonstrated the value of adversarial dataset construction as a methodology. Its eventual saturation by modern LLMs signals real progress in AI capabilities while showing the need for continuously evolving evaluation protocols.

The lasting contribution of HellaSwag is methodological rather than purely empirical. The adversarial filtering approach it introduced has become standard practice in benchmark creation, and its influence extends far beyond commonsense reasoning tasks.

The core insight, that the entities best positioned to identify exploitable patterns in a dataset are models trained to exploit it, represents a form of red-teaming that has permeated how the research community thinks about evaluation integrity. Instead of relying on human intuitions about what makes a benchmark hard, adversarial filtering uses empirical evidence: if a model trained for a few epochs achieves 80% accuracy, there are clearly artifacts to remove. This shifts benchmark quality assurance from a qualitative, human-driven process to a quantitative, model-driven one.

Subsequent benchmarks have extended this approach in several directions. WinoGrande applied adversarial filtering to pronoun disambiguation tasks, creating a large-scale version of Winograd schema challenges that resists statistical shortcuts. PIQA tested physical intuition using question-answer pairs filtered adversarially. Each of these borrowed the core adversarial filtering mechanism while adapting it to different reasoning domains.

The benchmark also played a useful role in establishing the scaling approach. The large jump in HellaSwag performance between GPT-2 and GPT-3 scale models, from around 48% to nearly 80%, was one of several results that motivated the scaling hypothesis: the idea that increasing model size and training data would eventually produce qualitatively new capabilities. HellaSwag was hard enough to discriminate between model generations in a way that simpler benchmarks could not. This makes it a useful signal in the early scaling literature.

Finally, HellaSwag's saturation has contributed to the ongoing conversation about benchmark validity and what it means for AI to "pass" a benchmark. The gap between 95% accuracy on HellaSwag and human-level physical commonsense in open-ended settings remains substantial. Models that score at human level on the benchmark still make errors on physical reasoning tasks outside the benchmark's specific distribution, still struggle with novel objects and unusual contexts, and still cannot ground their language understanding in sensorimotor experience the way humans can. This gap between benchmark performance and real capability has motivated a broader rethinking of what evaluation should measure and how benchmarks should be designed to remain informative as model capabilities continue to advance.

HellaSwag established a new standard for commonsense reasoning evaluation by introducing adversarial filtering, an iterative process that uses machine learning models to remove examples solvable through spurious statistical patterns. This approach created a multiple-choice completion task requiring physical reasoning rather than lexical pattern matching, successfully resisting the superficial shortcuts that plagued earlier natural language inference datasets.

Key takeaways include:

• Task design: HellaSwag tests commonsense reasoning through everyday activity completion, requiring models to understand physical causality, object permanence, and procedural coherence. The task requires simulating the physical world described in text rather than merely matching surface patterns between context and ending.

• Adversarial filtering: The dataset construction process iteratively trains discriminator models to find shortcuts in candidate examples, then generates harder distractors until the dataset resists known artifacts. This creates a computational arms race that produces reliable evaluation data while giving no formal guarantee against undiscovered shortcuts.

• Evaluation standard: Multiple-choice accuracy is the metric, with human performance (95%) giving an upper bound and random chance (25%) giving a lower bound. The large initial gap between early model performance (33%) and human performance demonstrated the effectiveness of the adversarial construction approach.

• Scoring approaches: Autoregressive models use conditional log-probability scoring with length normalization, while discriminative models use classification-based approaches. Length normalization is important for fair comparison across endings of different lengths.

• Saturation dynamics: Modern large language models have approached or reached human-level performance on HellaSwag, illustrating both the rapid progress in AI capabilities and the ephemeral nature of static benchmarks. This saturation motivates continuous benchmark renewal and raises important questions about data contamination and what exactly has been learned.

• Methodological legacy: The adversarial filtering methodology pioneered by HellaSwag has become standard practice in benchmark creation, influencing subsequent evaluations across mathematical reasoning, factual consistency, and physical intuition. The benchmark's approach to treating model failures as a diagnostic signal for dataset artifacts represents a lasting contribution to evaluation science.

HellaSwag's legacy comes less from the specific dataset, which has largely been saturated, and more from the methodology it pioneered. Adversarial filtering has informed subsequent evaluations like GSM8K for mathematical reasoning and TruthfulQA for factual consistency. As models continue to improve, the arms race between benchmark creators and model capabilities drives us toward increasingly sophisticated evaluation paradigms that separate pattern recognition from understanding of the physical and social world.

Looking back from the vantage point of several years later, HellaSwag's greatest contribution may be the demonstration that benchmark construction is itself a research problem. Creating a good benchmark means identifying an important capability to measure and actively engineering the evaluation to resist the specific failure modes of the models being evaluated. The HellaSwag authors showed that applying machine learning techniques to the benchmark construction process could produce substantially more reliable evaluations than human design alone. This insight, that models should be involved in building the benchmarks they will be tested on, has become a foundational principle of modern evaluation methodology. The next generation of benchmarks, whether they test reasoning, factual knowledge, coding ability, or alignment with human values, will likely incorporate adversarial construction techniques inspired by HellaSwag's approach, adapting them to the specific challenges of each new domain.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about HellaSwag and adversarial filtering.