HellaSwag › Worked Example

Walking through a concrete HellaSwag example makes the reasoning requirements tangible and reveals why certain distractors are particularly challenging. Let us examine an example that illustrates the different types of physical commonsense the task demands.

Context:

"A person is seen standing in a kitchen holding a knife and an orange. They begin by cutting the orange in half. They then take one half and begin slicing it into smaller pieces. They place the sliced orange pieces into a bowl."

Candidate Endings:

1. "They then take the other half and continue slicing it into pieces, adding them to the bowl."
2. "They then pour the orange juice into a glass and drink it."
3. "They then put the knife in the refrigerator to keep it fresh."
4. "They then use the knife to stir the orange slices in the bowl."

Analysis:

Option 1 is the gold ending. It continues the established activity logically, using the "other half" that was implicitly left over when the first half was sliced. It maintains the physical setup (knife, bowl), extends the pattern established by the previous actions (slicing pieces into the bowl), and completes the obvious goal of processing the entire orange. Notice how this ending requires understanding that "the other half" refers to the half not yet processed, a form of object tracking that requires understanding the physical state left by the preceding actions.

Option 2 describes a plausible outcome in some contexts, since oranges can be juiced. However, it introduces a glass that was never mentioned in the context, and it skips the necessary squeezing or juicing step that would be required to produce liquid from sliced pieces. More subtly, the context describes a slicing procedure, and the transition to pouring liquid juice is physically inconsistent with having produced solid sliced pieces rather than squeezed juice. This distractor tests whether models track the physical state of the orange (now sliced into solid pieces) rather than just recognizing that orange juice is a common outcome of working with oranges.

Option 3 violates physical commonsense entirely: knives are not refrigerated to keep them fresh, and refrigeration is not a step in orange preparation. This distractor maintains grammatical correctness and uses objects from the context (the knife), but describes an action that contradicts basic knowledge of kitchen practices and the preservation of metal tools. It is the easiest distractor to reject for a model with any physical world knowledge, but it is a useful sanity check.

Option 4 tests knowledge of tool affordances: knives are not used for stirring, and attempting to stir with a knife blade would be awkward and potentially dangerous. This distractor maintains both objects from the context (knife, bowl) and describes an interaction between them, which makes it superficially plausible to a model that tracks object co-occurrence without understanding functional properties. A model that merely checks "does this ending mention objects from the context?" would be equally drawn to Options 1 and 4, requiring physical knowledge about tool use to distinguish them.

The key insight from this analysis is that each distractor tests a different dimension of physical commonsense. Option 2 tests object state tracking, Option 3 tests basic physical plausibility, and Option 4 tests tool affordances. The adversarial filtering process is designed to ensure that distractors of this caliber, which require physical reasoning to reject, dominate the final dataset rather than distractors that can be rejected through simple lexical or syntactic cues.

This layering of different reasoning requirements across distractors is not accidental. The HellaSwag construction process generates multiple candidate distractors per example and selects the hardest surviving ones after filtering. The result is that each example often contains at least one distractor that is close to the gold ending in every measurable surface feature except physical plausibility, forcing models to engage with the substance of the physical scenario rather than its description.