HellaSwag › Evaluation Methodology

HellaSwag employs a straightforward multiple-choice accuracy metric, but the interpretation of this metric requires understanding the baseline performance levels and the significant gap between random guessing and human performance. This gap is a measure of the task's inherent difficulty and the effectiveness of the adversarial filtering process. Unlike benchmarks where near-ceiling human performance suggests the task is too easy, HellaSwag's design deliberately targets the large initial gap between model and human performance as evidence that the adversarial approach succeeded.

The choice of multiple-choice format over open-ended generation is deliberate and consequential. Open-ended generation would require a reference-based metric or human evaluation to assess whether the model's continuation is physically coherent, and both options introduce noise: reference-based metrics penalize valid continuations that differ from the reference, while human evaluation is expensive and slow. Multiple-choice format allows fully automatic evaluation: accuracy is deterministic and requires no human judgment once the dataset is constructed. This design choice trades off some generative capability measurement for scalability and reproducibility, a trade-off that has defined the mainstream of NLP evaluation since the rise of datasets like SNLI. HellaSwag inherits this trade-off deliberately, prioritizing the ability to run large-scale systematic evaluations over the richer information that free-form generation assessment would provide.

Formally, given a context $c$ and a set of four candidate endings $\{e_1, e_2, e_3, e_4\}$ where $e^*$ denotes the gold ending, a model assigns a score $s(c, e_i)$ to each candidate. The predicted ending is:

$$\hat{e} = \arg\max_{i \in \{1,2,3,4\}} s(c, e_i)$$

where the score function $s$ depends on the model architecture and evaluation strategy.

The overall accuracy on the test set of $N$ examples is:

$$\text{Acc} = \frac{1}{N} \sum_{j=1}^{N} \mathbf{1}[\hat{e}_j = e^*_j]$$

where:

• $N$: the total number of evaluation examples (10,042 in the validation set)
• $\hat{e}_j$: the model's predicted ending for example $j$
• $e^*_j$: the gold ending for example $j$
• $\mathbf{1}[\cdot]$: the indicator function, equal to 1 when the condition is true and 0 otherwise

Different model architectures compute the score $s(c, e_i)$ differently, which reflects their underlying training objectives and inductive biases:

• Masked language models (BERT): Concatenate context and ending with a [SEP] token, use the [CLS] representation as input to a linear classifier, or compute pseudo-log-likelihood by masking and scoring each token in the ending. This approach treats the task as discriminative classification.
• Autoregressive models (GPT): Compute the conditional probability $P(e_i \mid c)$ by concatenating context and ending, then summing log probabilities of the ending tokens given the context. Let the context tokenize to positions $1, \ldots, L$ and the ending to positions $L+1, \ldots, L+M$. The score is:

$$s(c, e_i) = \sum_{t=L+1}^{L+M} \log P(x_t \mid x_1, \ldots, x_{t-1})$$

where $x_t$ is the token at position $t$. This approach treats the task as conditional language modeling rather than classification.

• Encoder-decoder models: Encode the context and decode the ending, scoring based on reconstruction probability or sequence-to-sequence alignment loss. The ending that the model assigns the highest generation probability to is selected as the predicted continuation.

An important subtlety arises with length normalization. Longer endings accumulate more log-probability terms and therefore tend to receive lower raw scores simply because they have more tokens to predict. To control for this, practitioners often normalize by ending length:

$$s_\text{norm}(c, e_i) = \frac{1}{M_i} \sum_{t=L+1}^{L+M_i} \log P(x_t \mid x_1, \ldots, x_{t-1})$$

where:

• $M_i$: the number of tokens in ending $e_i$
• $L$: the number of tokens in the context $c$
• $x_t$: the token at position $t$ in the concatenated sequence

Length normalization prevents the model from systematically preferring shorter endings regardless of their content, which would otherwise create an exploitable artifact in the evaluation itself.

Random chance accuracy on HellaSwag is 25%. This reflects the four-way multiple choice design. Early results established important reference points that demonstrated the effectiveness of adversarial filtering:

• Bag-of-words baseline: 31% accuracy, showing that lexical overlap between context and ending provides some signal but falls far short of solving the task.
• BERT-base (zero-shot): 33% accuracy, showing slight improvement over lexical methods but still approaching random chance despite BERT's strong general language understanding capabilities.
• GPT-2 (fine-tuned): approximately 48% accuracy, showing that larger autoregressive models capture more physical regularities but still fall well below human performance.
• Human performance: 95% accuracy, near-ceiling performance showing that the task is well-defined and solvable for humans with ordinary commonsense knowledge.

The massive gap between early model performance (33%) and human performance (95%), spanning 62 percentage points, indicated that the adversarial filtering had successfully removed superficial cues that might have allowed statistical models to shortcut the reasoning process. This gap became the target for subsequent research and model development, motivating work on better commonsense knowledge integration, larger pretraining corpora, and improved architectures.