Human Evaluation Design › Summary

Human evaluation design requires balancing methodological rigor with practical constraints. The key principles covered in this chapter include:

• Interface design should minimize cognitive load through progressive disclosure, clear visual hierarchy, and appropriate annotation paradigms (absolute rating for objective criteria, comparative ranking for subjective preferences). The interface should be invisible to annotators, letting them to focus entirely on the text being evaluated. Scale design, session length, and response latency all affect the quality of judgments collected.

• Task instructions must define criteria explicitly, give positive and negative examples, and be refined through pilot testing rather than treated as clear by default. Instructions are hypotheses that require validation through cognitive interviewing and ambiguity auditing. Borderline examples and explicit edge case handling are often more informative than ideal-case examples.

• Annotator selection involves tradeoffs between expertise (accuracy on specialized tasks) and diversity (generalizability to user populations), with qualification tests filtering out unsuitable workers. Who you choose to evaluate your model determines what values get encoded in your evaluation. Demographic homogeneity produces demographically narrow evaluations.

• Cost estimation must account for annotation time, replication needs (typically 3 annotators per item), platform fees, and qualification overhead, with power analysis so sufficient sample sizes to detect real effects. Pilot studies give empirical estimates of time and effect size that sharply improve budgeting accuracy.

• Reliability scales with annotator count following the Spearman-Brown prophecy formula, with diminishing returns beyond three to five annotators. The formula $\rho_n = n\bar{r} / (1 + (n-1)\bar{r})$ lets you calculate the reliability target before collecting data.

The choices you make in evaluation design, from the wording of instructions to the demographic mix of annotators, constitute an implicit theory of what "good" language model behavior looks like. These choices are not neutral technical decisions but value judgments that shape what gets measured and therefore what gets optimized. As we move forward to examine inter-annotator agreement in the next chapter, remember that agreement is only real if you first ensured that the task you're asking annotators to perform is worth agreeing upon.