Human Evaluation Design › Why Human Evaluation Is Hard to Do Right

Before diving into the mechanics of evaluation design, it helps to understand why human evaluation fails so frequently in practice. The challenges are not accidental; they stem from basic properties of human cognition and the mismatch between research goals and the way humans naturally process information.

The first challenge is that language quality is subjective along many dimensions. When we ask annotators to rate "fluency," different people hold different mental models of what that means. A linguist might penalize subtle subject-verb agreement errors that a general user never notices. A non-native speaker might rate a slightly unusual phrasing as disfluent while a native speaker finds it charming. This subjectivity is not a defect in the annotators; it shows real variation in how humans process and value language. Your task as an evaluation designer is to reduce this variation by specifying more precisely what you mean, while accepting that some irreducible subjectivity will always remain.

The second challenge is anchoring and order effects. Humans are not independent judges who evaluate each item in isolation. They build up mental reference points as they annotate, and those reference points color subsequent judgments. An annotator who begins with ten excellent responses will rate a mediocre response more harshly than if they had begun with ten poor responses. This anchoring effect creates systematic biases that depend on the order in which items are presented, an order you typically choose or randomize without thinking carefully about its consequences.

The third challenge is the expertise gap between researchers who design evaluations and the annotators who complete them. Researchers have deep context about what they are trying to measure, why a particular distinction matters, and what counts as a good example. Annotators usually have none of this context. Instructions that seem unambiguous to the researcher often contain assumptions that are opaque to the annotator. The researcher knows that "factual accuracy" means accuracy relative to verifiable external sources, not internal consistency; the annotator may not. Bridging this gap requires more explicit communication than most researchers assume.

Understanding these failure modes shapes every design decision we discuss in this chapter. When we recommend progressive disclosure in interface design, we are compensating for limited working memory. When we recommend negative examples in instructions, we are working with the human tendency to learn from contrast. When we recommend pilot testing, we are acknowledging the expertise gap. Good evaluation design is applied cognitive science.