Benchmark Design › What a Benchmark Is
Language AI HandbookAll chapters Benchmark Design › What a Benchmark IsOne section · Sections in this chapter2 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before diving into construction, it is worth being precise about terminology. In NLP and LLM evaluation, the word "benchmark" is used loosely to mean many different things: a leaderboard, a dataset, a task, a paper, or a collection of all of these. Getting precise helps you think clearly about what you are building and what decisions matter.
A benchmark is a standardized evaluation suite consisting of one or more tasks, a dataset for each task, evaluation metrics, and a protocol specifying how models should be evaluated and compared. The distinguishing feature is that a benchmark is reusable: many different systems can be evaluated against it, letting comparison.
A single benchmark can contain one task (like SQuAD for reading comprehension) or dozens of tasks (like BIG-Bench with 204 tasks, or MMLU with 57 subject areas). The multi-task framing is increasingly common because no single task can adequately characterize a general-purpose language model. A model that scores 90% on question answering might score 60% on commonsense inference; a model that excels at factual recall might struggle with multi-step reasoning. Multi-task benchmarks expose this heterogeneity and allow you to reason about which capabilities are well-developed and which need work.
A benchmark is distinct from a "test set," which typically refers to the held-out split used during a single model's development. A test set is usually specific to one project and one model family; a benchmark is designed to be model-agnostic and community-wide. A benchmark is also distinct from a "dataset," which is just the data: a benchmark adds the evaluation protocol, the metrics, and often a leaderboard that makes comparison possible. The protocol matters more than you might expect. Two researchers using the same dataset but different prompting strategies, tokenization conventions, or score aggregation methods can produce incomparable numbers, as happened repeatedly in early large language model evaluation before evaluation harnesses like EleutherAI's lm-evaluation-harness standardized these choices.