HellaSwag › Key Evaluation Parameters
Michael BrenndoerferFebruary 28, 20261 min read
Language AI HandbookAll chapters Part 54 of 67Benchmark Evaluation
HellaSwag › Key Evaluation ParametersOne section · Sections in this chapter8 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Understanding the configurable aspects of HellaSwag evaluation helps interpret results across different settings and ensures comparisons are meaningful:
- model_name: The pretrained model identifier (for example, "gpt2", "gpt2-xl", "meta-llama/Llama-2-7b"). Larger models generally perform better but require more computation and memory.
- max_length: Maximum token length (commonly 512 or 1024) for context and ending concatenation. Longer sequences are truncated. Truncation can affect results if contexts exceed the limit, though this is rare given the typical context length.
- length_normalization: Whether to normalize log probabilities by ending length. This is recommended to avoid systematic bias toward shorter endings, but some published results omit it.
- max_examples: Number of examples to run. The full validation set contains 10,042 examples. Smaller samples introduce sampling variance, so results on fewer than 500 examples should be interpreted with caution.
- device: Computation device ("cuda" or "cpu"). GPU acceleration significantly speeds up inference for neural language models.
- random_baseline: Expected accuracy for uniform random guessing is 0.25, since there are four options by construction.
- human_performance: Human annotators achieve approximately 95% accuracy, serving as the practical upper bound for the benchmark.