HellaSwag › Dataset Construction in Detail

HellaSwag sources its contexts from ActivityNet, a large-scale video understanding dataset containing human-annotated descriptions of everyday activities. These captions provide naturally occurring descriptions of procedural knowledge: how to make a sandwich, change a tire, or perform a yoga pose. The video origin is important because it grounds the text in physical reality; the gold endings describe actions that occurred and were visually verified by human annotators who watched the video footage. This connection to physical reality ensures that the "correct" answer corresponds to observed events in the world that a human observer confirmed by watching them happen, rather than a matter of opinion or stylistic preference.

The ActivityNet dataset contains approximately 20,000 videos spanning 200 activity categories, with each video accompanied by dense temporal annotations describing what happens at different points in the activity. HellaSwag selects from this pool using criteria designed to ensure the completion task remains well-defined and challenging.

Contexts are selected to satisfy several strict criteria that together ensure the task remains well-defined and challenging:

• Temporal coherence: The context must describe a continuous activity without jumps in time or location. The narrative should flow logically from one action to the next, establishing a clear physical trajectory that the ending must continue.
• Physical grounding: The activity must involve observable physical actions rather than abstract reasoning or emotional states. This ensures that the task tests physical commonsense rather than theory of mind or emotional intelligence, keeping the evaluation focused on a specific and tractable form of reasoning.
• Completion ambiguity: The context must be sufficiently open-ended that multiple continuations appear plausible to a reader who has not seen the video, but sufficiently constrained that physical constraints eliminate most physically impossible options. If only one continuation is remotely plausible, the task becomes trivial. If too many are plausible, the task becomes ambiguous.

The average context length of 35-40 words provides enough detail to establish the physical situation without revealing the ending. This length was determined empirically through pilot studies: shorter contexts allowed too many plausible continuations, making the task ambiguous even for humans, while longer contexts made the task trivial by revealing the outcome or the specific objects involved in the final steps.

This balance between ambiguity and constraint is a subtle design achievement. Consider what happens at the extremes. A one-sentence context such as "A person holds a hammer" is too underspecified: hundreds of activities involve hammers, and the correct continuation could be any of them. A ten-sentence context that describes an entire nailing procedure in complete detail is so specific that only one continuation is plausible for anyone who has ever used a hammer. The 35-40 word sweet spot preserves just enough ambiguity that naive pattern-matching fails while giving just enough specificity that the correct continuation is uniquely determined by physical reasoning. Finding this balance required iterating over many potential context lengths with human judges, adjusting until inter-annotator agreement on the correct ending was consistently high while model performance remained near chance.

The adversarial generation employs three complementary strategies to create challenging distractors that resist simple statistical detection. Each strategy targets a different dimension of physical reasoning. This keeps the dataset tests multiple types of commonsense knowledge.

Generative model sampling uses language models to produce completions conditioned on the context. Early HellaSwag construction used GPT-1 and GPT-2 scale models as generators. These models produce plausible-sounding narratives but frequently hallucinate objects or actions inconsistent with the physical setup described in the context. A generated distractor might use a tool that was not mentioned, reference a material that has different properties than implied, or describe an action in the wrong order. After adversarial filtering, the distractors that survive are those that are plausible enough to fool a discriminator while still being incorrect.

Counterfactual editing involves modifying gold endings by changing key actions or objects. Substituting "pour" for "stir," or replacing "bowl" with "bag," creates physically impossible but grammatically coherent scenarios. These edits test whether models understand the functional affordances of objects: what a knife can and cannot do, what a bag can and cannot hold, which actions are reversible and which are not. Counterfactual editing is particularly effective at targeting models that reason about physical properties, because the edited distractors preserve every surface feature of the correct ending except the specific word that encodes the physical constraint.

Retrieval from similar contexts takes endings from different but semantically similar contexts and presents them as options for the target context. An ending about "baking the mixture" might be retrieved for a context that only mentions cold preparation, or an ending about "inserting screws" might be retrieved for a context involving adhesive assembly. These retrieved distractors are physically plausible in the abstract but violate the specific constraints established by the context. They test whether models track the state of the physical world described in the context rather than just recognizing that the ending describes a physically possible action in general.

Each generated distractor undergoes automatic filtering to ensure grammatical correctness and surface plausibility before entering the adversarial filtering loop. This pre-filtering ensures that the adversarial process focuses on semantic and physical coherence rather than basic linguistic validity. A distractor that is grammatically incoherent is trivially rejected by any model; the interesting distractors are those that are grammatically perfect but physically wrong.