Human Evaluation Design › Selecting Annotators

Who evaluates your model shapes what you measure. The "average" human evaluator is a statistical fiction; demographic factors, domain expertise, and linguistic background all influence judgments. When you select annotators, you are implicitly defining the population for whom your model should be optimized. A model evaluated solely by young, technically skilled, American English speakers will be optimized for that demographic, potentially failing for older users, non-technical audiences, or speakers of other English varieties.

The choice between domain experts and crowd workers (via platforms like Amazon Mechanical Turk or Scale AI) depends on your evaluation goals. This decision involves basic tradeoffs between accuracy, generalizability, and cost that have no universal correct answer.

Expert annotators (linguists, doctors, lawyers, software engineers) give:

• Higher accuracy on specialized tasks (medical question answering, legal reasoning, code generation)
• Better detection of subtle errors that untrained annotators miss
• Consistent application of complex criteria requiring domain knowledge

But experts are expensive ($50-$200/hour), scarce, and may have idiosyncratic standards that don't generalize to typical users. They may also be too forgiving of domain-specific errors because they mentally correct them automatically, or they may apply standards that are too high for general use. An expert programmer might penalize code that uses suboptimal algorithms even when that code works correctly for most users' needs. Expert annotators also tend to be more opinionated about style, which can introduce systematic biases when evaluating creative tasks.

Crowd annotators give:

• Demographic diversity that matches your user base
• Scalability for large datasets
• Lower cost ($12-$25/hour for US workers, less internationally)

However, crowd workers may lack domain knowledge, require extensive training, and vary in English proficiency and cultural background. They may miss subtle errors that domain experts catch immediately, or they may apply inconsistent criteria when evaluating complex outputs. The variability in crowd worker quality means you must invest more effort in screening and quality control. You also get less context about individual annotators, which makes it harder to diagnose systematic biases in specific workers.

For most general language model evaluation, a hybrid approach works best: use crowd workers for broad preference judgments and expert annotators for error analysis and safety-necessary applications. As we saw in Part XXXVII regarding human preference data collection, modern alignment pipelines often use crowd workers for initial preference labeling, then validate subsets with experts. This approach balances scale with accuracy, using experts where their judgment is needed and crowds where demographic diversity matters more than specialized knowledge.

Language models serve global populations, but evaluation datasets often reflect narrow demographics. This homogeneity creates blind spots where models perform well for the evaluated population but poorly for others. Consider the following dimensions of demographic variation:

Native speaker status: Non-native speakers may rate fluent but nonsensical text higher than native speakers because they attribute confusion to their own language skills rather than model incoherence. Conversely, native speakers might penalize grammatical variations that are acceptable in World Englishes. If your model serves global users, your evaluation pool should include both native and fluent non-native speakers in proportions that reflect your actual user base.

Cultural background: Politeness norms vary materially across cultures. A direct refusal might rate as "rude" in East Asian contexts but "honest" in Germanic contexts. Indirectness valued in high-context cultures may read as evasive to annotators from low-context backgrounds. If your model serves multiple cultures, your evaluation pool should reflect that diversity rather than defaulting to Western communication norms as the universal standard.

Age and technical familiarity: Younger, more technically advanced annotators have different expectations of AI systems than older, less technical users. They may accept awkward phrasing that technically communicates the information, while older users may find the same response confusing or off-putting. If your model serves a broad age range, ensure your annotator pool spans that range as well.

Subject matter expertise: When evaluating RAG systems from Part XLIV, annotators need enough domain knowledge to verify factual claims. A layperson cannot evaluate whether a medical explanation is accurate, even if it sounds plausible. Similarly, evaluating code generation requires programming ability, and evaluating legal reasoning requires legal training. Mismatch between annotator expertise and task requirements produces noise that looks like disagreement but reflects knowledge gaps.

The practical implication is that you should explicitly plan annotator demographics before recruiting. Define your target annotator profile the way you would define a target user persona, and actively recruit to reach that profile rather than accepting whoever happens to accept your task. Most annotation platforms allow you to filter by location, device type, and previous task performance; use these filters to approximate your desired demographic mix.

Not every annotator is suitable for every task. Implement qualification tests that sample the difficult cases your instructions address. A qualification test is a filter that ensures only annotators able to your specific task enter your dataset.

A qualification test for summarization evaluation might include:

• A summary containing a subtle factual error (testing attention to detail)
• A coherent but irrelevant summary (testing relevance criteria)
• A grammatically flawed but accurate summary (testing whether annotators conflate style with substance)
• An excellent summary that meets all criteria (establishing the standard for the highest rating)

Require 80-90% agreement with gold labels to pass. This threshold filters out random guessing while accounting for legitimate ambiguity. If you set the threshold too high (e.g., 100%), you eliminate qualified annotators who happen to disagree on ambiguous cases; if you set it too low (e.g., 50%), you admit annotators who are guessing randomly. The qualification test should take no longer than 10-15 minutes, as longer tests reduce the number of workers who attempt them and can create selection effects.

Monitor annotator performance continuously using embedded gold standards (items with known correct answers). Remove annotators who fall below threshold, but first verify that the disagreement stems from annotator error rather than instruction ambiguity. If multiple annotators fail the same gold standard item, the item might be mislabeled or your instructions might be unclear for that case type. Continuous monitoring protects your dataset from drift in annotator quality over time, as annotators may become fatigued or less careful as they work through large batches.

Pay qualification fairly, even for annotators who do not pass. Requiring unpaid work for qualification tests creates adverse selection: only annotators who are desperate for work complete unpaid tests, which is exactly not the population you want. Paying for qualification tests, even at a lower rate than the main task, ensures you draw from a wider pool and signals respect for annotators' time that tends to attract more conscientious workers.

Every annotator pool encodes the biases of the people who comprise it. When you recruit exclusively from platforms that over-stand for certain demographics, your evaluation will reflect those demographics' values and preferences. This creates both an accuracy problem and a fairness problem, because models optimized for these evaluations will perform better for the represented groups and worse for others.

Some biases are systematic and predictable. Annotators from high-income countries tend to rate AI systems that are more deferential and formal as higher quality, while those from other contexts may prefer directness. Annotators with higher education levels tend to penalize grammatical errors more harshly. These are not random noise; they are structured biases that can systematically tilt your evaluation in predictable directions.

The appropriate response is not to eliminate biases (which is impossible) but to make them explicit and manage them. Document your annotator demographics as carefully as you document your data. When reporting evaluation results, specify who evaluated your model, just as clinical trials specify the patient population. This transparency allows others to contextualize your results and assess whether your evaluation is needed to their use case.