Benchmark Design › Step 1: Task Definition and Scope

Every benchmark begins with a question: what capability are you trying to measure? This sounds simple, but operationalizing an abstract capability into a concrete, measurable task is one of the hardest parts of benchmark design. The history of NLP evaluation is full of cases where researchers thought they were measuring one thing and discovered, after careful analysis, that their benchmark was measuring something quite different.

Suppose you want to measure "commonsense reasoning." This is a meaningful goal, but it is underspecified in ways that matter. Commonsense reasoning could mean:

• Physical intuition: objects fall when dropped, water is wet, you cannot walk through walls
• Social reasoning: knowing that someone who frowns is probably unhappy, or that refusing a gift is sometimes polite
• Temporal reasoning: events have causes that precede them, processes take time
• Spatial reasoning: objects cannot be in two places at once, putting a lid on a box encloses its contents
• Causal reasoning: understanding why events happen and what would follow if conditions changed

Each of these is a distinct sub-capability, and a model might excel at one while failing at another. GPT-2 era models could handle many physical intuition examples but struggled with multi-step social reasoning. Current large models pass most commonsense benchmarks but still fail in systematic ways on physical intuition tasks that require composing multiple constraints simultaneously. The first decision in benchmark design is how broad or narrow to make the scope. Broader scope gives you a more complete picture but harder-to-diagnose failures; narrower scope gives you cleaner signal but may miss important facets of the target construct.

A useful principle is to ask: "What would it mean to pass this benchmark completely?" If a model scores 100% on your benchmark but you still have serious doubts about whether it has the target capability, your benchmark is not well-targeted. Conversely, if a model scoring near-chance would be convincing evidence of complete incapability, you have a tightly defined target. Running this thought experiment before you collect a single example will save you from discovering the mismatch only after months of annotation work.

This question also forces you to confront the competence-performance distinction. A model might possess a capability in some sense but fail to demonstrate it in your task format due to framing effects, prompt sensitivity, or evaluation artifacts. Benchmark design tries to minimize the gap between latent competence and measured performance, but it can never eliminate it entirely. Acknowledging this gap upfront shapes how you interpret scores and motivates the validation steps described later.

Once you have a target capability, you need to choose a task format that elicits it. The format choice is consequential: different formats favor different solution strategies, and some strategies align with the target capability while others do not. Common formats include:

• Multiple-choice question answering (MCQA): The model selects from 4-5 options. Easy to score automatically, but susceptible to option-order bias and guessing.
• Open-ended generation: The model produces free-form text. Rich but difficult to evaluate; requires human judges or a secondary model.
• Reading comprehension (extractive): The model extracts a span from a given document. Natural and precise, but limited to information present in the document.
• Natural language inference (NLI): The model classifies a hypothesis as entailed, contradicted, or neutral given a premise. Tests logical and linguistic reasoning.
• Cloze / fill-in-the-blank: The model predicts a masked token or phrase. Tightly controlled but feels artificial.
• Dialogue and conversation: The model generates conversational responses. Hard to score automatically; captures pragmatic and social reasoning.
• Ranking and preference: The model ranks candidate outputs from best to worst. Natural for tasks where quality is comparative rather than absolute.

The format choice creates important tradeoffs. Multiple-choice is popular because it is cheap to score and has a clear chance baseline ($1/k$ where $k$ is the number of options). A 4-choice benchmark has a 25% chance baseline, a 5-choice benchmark has 20%. This gives you an interpretable lower bound: any model that does not exceed chance either has no relevant capability or cannot express it through the task format.

But MCQA has well-documented failure modes. Models can exploit option statistics, such as the fact that correct answers are often longer or more specific than distractors. They can exploit the framing of wrong answers, which annotators often construct by systematic negation or minor modification of the correct answer. They can be sensitive to the ordering of choices, performing better when the correct answer appears in early positions. Careful benchmark design counteracts these failures, for instance by so distractors are plausible and matched in style and length to correct answers, and by randomizing option order.

Open-ended generation is richer but requires either expensive human evaluation or proxy metrics like BLEU, ROUGE, or BERTScore. As discussed in the Evaluation Fundamentals chapter, these automated metrics are imperfect approximations of human judgment. BERTScore correlates better with human ratings than BLEU for most tasks, but even BERTScore misses important aspects of quality like coherence, factual accuracy, and appropriate register. For benchmarks where the quality of generated text is the core measure, this is a real limitation rather than an engineering inconvenience.

The choice between formats should be driven by the target capability. If you are measuring factual recall, MCQA is appropriate and efficient. If you are measuring coherent argumentation or fine-grained explanation, you probably need open-ended generation or at least a free-response format with structured evaluation criteria. Forcing a complex generative capability into a multiple-choice frame often degrades ecological validity, even if it improves measurement precision.

A proxy trap occurs when the task format measures something other than the target capability. For example, if you design a reading comprehension task where the answer is always a named entity, you may be testing named entity recognition rather than comprehension. If your commonsense reasoning questions can all be answered by statistical associations in text (e.g., "sky is [MASK]" $\to$ "blue"), you are measuring distributional co-occurrence rather than reasoning. If your mathematical reasoning task always has answers that are single-digit integers, a model that only knows common integers will perform above chance without any mathematical ability.

Proxy traps are insidious because they are hard to see from inside the design process. The benchmark creator is immersed in the task domain and may not notice that certain strategies they would never use are effective. The standard remedy is adversarial testing: before finalizing a benchmark, deliberately try to solve it without using the target capability. Write a simple program that uses only surface features (answer length, option frequency, lexical overlap with the question). If this program performs substantially above chance, you have a proxy trap.

Checking for proxy traps requires adversarial thinking: ask "what strategy other than the target capability would allow a system to do well?" Then systematically eliminate those strategies, or at minimum measure them separately so you can account for them in score interpretation. This is not a one-time check. As the field adapts to your benchmark, new proxy strategies will emerge. Monitoring whether performance improvements correlate with capability improvements rather than strategy adaptations is an ongoing responsibility.