Human Evaluation Design › Crafting Task Instructions

Instructions are the contract between you and the annotator. Ambiguity in instructions doesn't average out across annotators; it compounds into noise that masks real signal. When two annotators interpret the same instruction differently, their disagreement is not real difference in perception but failure of communication. Writing effective instructions requires anticipating the edge cases and interpretations that diverge from your intent. You must imagine how someone unfamiliar with your research goals might understand your words, and you must explicitly close off misinterpretations that seem obvious to you but invisible to others.

Effective instructions operate at multiple levels. This hierarchy ensures that annotators understand the actions they need to take and the conceptual framework underlying your evaluation. By giving context at each level, you let annotators to handle novel situations that you haven't explicitly covered in examples.

1. The Task Definition (The "What")

Begin with a one-sentence description of what the annotator will do. For example: "You will read short conversations between a user and an AI assistant, then rate the assistant's final response for helpfulness."

This sentence should name the input (what they will read), the output (what they will produce), and the criterion (what they are judging). Omitting any of these three components creates ambiguity. "Rate these AI responses" leaves open what constitutes a response, what they are rating them against, and what "rate" means operationally.

2. The Evaluation Criteria (The "How")

Break down abstract qualities into observable behaviors. Instead of "Rate helpfulness," specify:

• A helpful response directly addresses the user's question
• A helpful response gives sufficient detail without being verbose
• A helpful response acknowledges uncertainty rather than hallucinating facts

These behavioral anchors translate abstract concepts into concrete actions that annotators can recognize in text. Without such specification, annotators must rely on their own idiosyncratic definitions of "helpfulness," which vary widely across individuals and cultures. The criterion of acknowledging uncertainty is particularly important for language model evaluation: a response that confidently gives wrong information is less helpful than one that says it is unsure, even though the latter may feel less satisfying to read.

3. The Scale Anchors (The "How Much")

Define each point on your rating scale with concrete examples. For a 5-point helpfulness scale:

• 5 (Excellent): The response fully answers the question, gives needed context, and anticipates follow-up needs.
• 4 (Good): The response answers the question clearly with adequate detail, with minor gaps that don't affect usefulness.
• 3 (Adequate): The response answers the core question but lacks detail or misses minor nuances.
• 2 (Poor): The response partially answers the question or contains significant gaps.
• 1 (Unacceptable): The response is irrelevant, incomplete, or actively misleading.

Scale anchors serve as reference points that calibrate annotators to your intended meaning. They prevent the compression or expansion of scales that occurs when different annotators interpret "good" or "bad" differently. By describing what constitutes a 5 versus a 3 versus a 1, you create a shared metric that remains stable across annotators and over time. Without explicit anchors at every scale point, annotators tend to cluster responses in the middle of the scale, avoiding extremes because they are uncertain what extreme ratings mean.

4. Edge Cases (The "What If")

Explicitly address common ambiguities:

• If the user asks something harmful, rate whether the refusal is appropriate, not whether it answers the question.
• If the response contains code, verify it works conceptually; you don't need to run it.
• If the response says "I don't know," rate whether the uncertainty is warranted given the query.

Edge cases often constitute the most challenging and informative examples in your dataset. Without guidance, annotators may apply inappropriate criteria (e.g., penalizing a model for refusing to generate hate speech) or waste time deliberating over decisions you could have anticipated. Proactive handling of edge cases shows respect for annotators' time and ensures consistent handling of difficult scenarios. A good practice is to collect a sample of 50-100 items before writing your instructions, identify the most common ambiguous cases, and address each one explicitly. Your instructions should be a product of empirical observation, not abstract planning.

Humans learn more from seeing what not to do than from descriptions of ideal performance. Cognitive science research consistently shows that errorful examples highlight necessary features that positive examples obscure. When you see a correct summary, you might not notice all the ways it could have gone wrong; when you see an incorrect one, the specific failure mode becomes salient. Include counter-examples that illustrate common failure modes:

Notice how the second negative example catches a subtle error that annotators might miss if they only looked for obvious contradictions. This level of specificity calibrates annotators to your standards. The first negative example establishes that date errors matter; the second establishes that attributions must be precise. Together, they create a mental model of what constitutes "factual accuracy" that is richer than any definition could give.

Borderline examples are as important as clear positive and negative cases. If the boundary between a 3 and a 4 on your scale is where annotators tend to disagree, show examples that fall explicitly on each side of that boundary and explain why. The explanation of what distinguishes a 3 from a 4 is often more informative than the examples themselves, because it articulates the criterion in terms that annotators can apply to future cases that look different from your examples.

There is a tension in instruction design between completeness and readability. Longer instructions cover more edge cases but also require more time to read, and many annotators skim long instructions rather than reading them carefully. Research on annotation quality suggests that annotators who spend more time on instructions produce more consistent judgments, but only up to a point: instructions that take more than ten minutes to read see diminishing returns as annotators become fatigued before starting the task.

The practical resolution is to structure instructions hierarchically. Lead with the information needed to handle typical cases, then give supplementary sections that cover edge cases and detailed criteria, with examples. Annotators can read the core section, begin annotating, and return to the supplementary sections when they encounter confusing items. This structure respects the fact that reading instructions in the abstract is less effective than reading them in relation to concrete examples.

Consider presenting instructions alongside a tutorial. Rather than asking annotators to read instructions before seeing any data, walk them through five or six example items with explanations of the correct judgment and why. Tutorial-style onboarding, where annotators make judgments and receive immediate feedback before beginning the actual task, produces better calibration than instruction-only approaches. The interactive element helps annotators connect abstract criteria to concrete cases in a way that passive reading cannot.

Instructions are hypotheses about how to communicate your task. Like all hypotheses, they require testing. The first draft of instructions, no matter how carefully written, inevitably contains ambiguities and assumptions that only become apparent when actual humans attempt to follow them. Before launching a full evaluation:

1. Pilot with 3-5 examples: Have team members annotate a small batch using the instructions. Where do their judgments diverge from your intent? Internal piloting catches obvious errors in instruction logic and reveals where your examples don't match your criteria.

2. Cognitive interviewing: Ask pilot annotators to think aloud while reading instructions. Where do they pause, reread, or express confusion? This technique, borrowed from survey methodology, reveals the mental models annotators bring to your task. If they pause at a particular criterion, they may be searching their memory for how to apply it; if they reread, they may have encountered unfamiliar terminology.

3. Ambiguity auditing: Review disagreements between annotators. If two qualified annotators give different ratings for the same output, your instructions likely permit multiple valid interpretations. High disagreement on specific items often indicates that those items occupy a boundary case your instructions haven't adequately defined.

Refine instructions based on these findings. This iteration is not a luxury; it is a requirement for valid measurement. A study with well-tested instructions and 100 examples yields more reliable data than one with ambiguous instructions and 1,000 examples. The cost of iteration is small compared to the cost of collecting data you cannot interpret because you don't know what the annotators were measuring.

A final practical note: maintain version control of your instructions. When you revise instructions mid-study (which you should avoid but sometimes cannot), label the annotations by instruction version so you can account for any shifts in standards. Mixing annotations collected under different instruction versions can introduce systematic confounds that are difficult to detect after the fact.