Benchmark Design › Step 3: Annotation Guidelines

For any benchmark that requires human labeling, which describes most benchmarks, annotation guidelines are the document that bridges your intent and the annotators' behavior. Good annotation guidelines are the difference between a reliable benchmark and a noisy one. The investment in writing careful guidelines pays dividends throughout the annotation process: lower disagreement, faster annotation, fewer examples that need to be discarded or re-annotated.

Writing annotation guidelines is a skill that combines technical precision with instructional design. Guidelines must be clear enough that a worker with no prior NLP background can apply them consistently, yet complete enough to handle the edge cases that will inevitably appear. The principles of effective guidelines are:

Explicit over implicit. Do not assume annotators share your background knowledge or intuitions. State every decision rule in writing. If you think "obviously, the answer should be factual," make that criterion explicit with examples of what factual means and what it does not mean. The guideline writer's intuitions about what is "obvious" are almost always wrong about at least some edge cases.

Illustrated with examples. For every decision rule, show at least one positive example (an instance that passes the rule) and one negative example (an instance that fails it). Edge cases deserve their own examples, not a blanket "use your judgment" instruction. "Use your judgment" is not a guideline; it is an abdication of the guideline writer's responsibility.

Hierarchically ordered. When rules conflict, annotators need a clear priority ordering. For example: "If the question is ambiguous, prefer the more conservative interpretation. If both interpretations lead to the same answer, select either. If they lead to different answers, flag the example for review." Annotators who do not have a conflict resolution hierarchy will handle conflicts idiosyncratically, introducing systematic noise that is hard to detect or correct.

Revised through iteration. The first draft of annotation guidelines is never the final draft. You should run a pilot annotation round on a small set of examples, measure annotator disagreement, identify the sources of disagreement, and revise the guidelines to eliminate ambiguity. Repeat this cycle until disagreement is low. Most professional annotation projects run two to four pilot rounds before beginning full-scale annotation. Skipping this step is a common mistake that leads to discovering guideline problems only after thousands of examples have been labeled.

Accessible in format. Guidelines that are organized well, with a clear table of contents, consistent formatting, and searchable text, get used more reliably than dense prose documents. Annotators consult guidelines when they are uncertain; if the format makes it hard to find the relevant rule, they will guess instead. Short, modular guideline sections with clear decision flowcharts or tables are more useful than complete but impenetrable prose.

Inter-annotator agreement (IAA) is a measure of how consistently different annotators label the same examples. It is the main quality signal for annotation pipelines, and it serves two purposes: it quantifies the reliability of your annotations, and it identifies specific guideline weaknesses that are causing disagreement.

For categorical labels such as NLI entailment/contradiction/neutral, Cohen's Kappa ($\kappa$) is the standard measure. Raw percentage agreement is misleading because it does not account for the chance of two annotators independently selecting the same label. If there are only two label options and each annotator picks one uniformly at random, they will agree 50% of the time purely by chance, which should not count as real agreement. Cohen's Kappa corrects for this:

$$\kappa = \frac{P_o - P_e}{1 - P_e}$$

where:

• $P_o$: the observed proportion of agreement between annotators (how often they agreed)
• $P_e$: the proportion of agreement expected by chance, computed from the marginal label distributions of each annotator
• $1 - P_e$: the maximum possible improvement over chance agreement, which normalizes the numerator

To see how $P_e$ is computed: suppose annotator A labels 60% of examples as positive and 40% as negative, and annotator B labels 55% as positive and 45% as negative. By chance, both will agree on "positive" for $0.60 \times 0.55 = 0.33$ of examples and on "negative" for $0.40 \times 0.45 = 0.18$ of examples, giving $P_e = 0.51$. If observed agreement is $P_o = 0.80$, then $\kappa = (0.80 - 0.51) / (1 - 0.51) = 0.59$.

A $\kappa$ of 1.0 means perfect agreement; 0.0 means agreement no better than chance; negative values indicate systematic disagreement (rarer than chance). Common interpretations follow Landis and Koch (1977): $\kappa < 0.20$ is poor, $0.21$-$0.40$ is fair, $0.41$-$0.60$ is moderate, $0.61$-$0.80$ is substantial, $> 0.80$ is almost perfect.

For many NLP tasks, a $\kappa$ above 0.6 is considered acceptable; above 0.8 is high quality. If you are getting $\kappa < 0.5$, it usually means either the task itself is inherently ambiguous (which is important information: you may need to refine the task definition) or the guidelines need clarification. The distinction matters: if disagreement is concentrated on a specific category of examples, it suggests a guideline gap. If disagreement is spread uniformly across all examples, it suggests the task itself is more ambiguous than you intended.

For tasks where multiple annotators might all be correct (as in paraphrase generation, where many phrasings can be valid), strict inter-annotator agreement metrics are inappropriate. Fleiss's Kappa handles multiple annotators simultaneously, while Krippendorff's Alpha handles ordinal ratings and partial agreement. Choosing the right metric for the task type is part of benchmark design, not an afterthought.

After measuring IAA, you need a policy for resolving disagreements. The choice of policy shapes which examples end up in your benchmark and what the gold labels look like. The main options are:

Majority vote. Use the label agreed upon by a majority of annotators. Simple and intuitive, but majority can still be wrong if all annotators share the same misconception, particularly when annotators are recruited from a homogeneous population with shared biases. Majority vote works best with five or more annotators so that the majority is more likely to reflect a stable consensus rather than random variation.

Adjudication. A senior annotator or domain expert reviews all disagreements and makes a binding decision. More accurate than majority vote when domain expertise is available, because the adjudicator can reason about edge cases and consult reference material. Expensive in both time and cost, but often necessary for specialized domains or high-stakes benchmarks. Several prominent benchmarks like HellaSwag and WinoGrad used expert adjudication for difficult cases.

Exclude ambiguous examples. Simply drop examples with high annotator disagreement. This improves benchmark cleanliness but may introduce selection bias toward unambiguous cases. If the most interesting or challenging examples are also the most ambiguous, excluding them produces a benchmark that underrepresents exactly the edge cases where model capability is most in question. Use this strategy deliberately and document it.

Preserve the disagreement. Some benchmarks like ChaosNLI deliberately preserve the distribution of human judgments rather than collapsing to a single gold label. When there is real human disagreement about the correct label, collapsing to a single "gold" label suppresses information about task ambiguity that is itself scientifically interesting. Preserving distributions allows you to measure how well models predict human agreement patterns, which may be more informative than binary correct/incorrect scoring for intrinsically subjective tasks.

The right policy depends on your benchmark's purpose. For a benchmark intended to measure model-human alignment on subjective judgments, preserving disagreement is the most faithful approach. For a benchmark intended to measure factual correctness with an objective ground truth, adjudication by domain experts is preferable. For most standard capability benchmarks, majority vote with high annotator count (five or more) is a practical default.