Instruction Following Evaluation › Human Evaluation

Human evaluation remains the gold standard for assessing instruction following. Asking people gives the most direct answer about whether a response answers accurately and is useful in its context. If we want to build models that satisfy users, their judgment is the ultimate measure of success. However, human evaluation introduces its own complexities that practitioners must understand and handle carefully.

Think of human evaluation as commissioning a review of a restaurant by professional food critics. Their assessments are authoritative and fine-grained, capturing dimensions that no automated system can fully replicate: whether the ambiance matched the occasion, whether the waiter's recommendations were trustworthy, whether the portion sizes felt appropriate for the price. But professional reviews take time and money, and each one reflects the critic's particular palate and expectations. Scaling to thousands of reviews per week requires a different approach.

The basic trade-off in human evaluation is between quality and scale. Individual expert judgments are high-quality but expensive and slow. Crowdsourced judgments from platforms like Amazon Mechanical Turk scale readily but require careful quality control because worker incentives and expertise vary. Arena-style judgments from real users scale naturally and reflect authentic preferences, but the data arrives slowly and unevenly across model pairs.

The most common human evaluation protocol presents evaluators with two responses to the same instruction and asks them to select the better one. This approach uses a basic insight from psychology: humans are materially better at making relative comparisons than absolute judgments. When asked to rate a response on a 1-10 scale, different evaluators may have wildly different internal calibrations, and even the same evaluator may shift their standards over the course of a long evaluation session. But when asked which of two responses is better, people tend to agree more consistently:

In this example, Response A gives a scientific explanation, while Response B uses an intuitive analogy. The "better" response depends on the user's intent. This shows the subjectivity of the task. If the instruction had specified "explain to a child," most evaluators would prefer Response B. Without that specification, different evaluators may legitimately reach different conclusions based on their assumptions about the target audience. This is not a failure of the evaluation protocol; it is an accurate reflection of the real ambiguity in what "good" means for open-ended instructions.

Pairwise comparison has several advantages. It is cognitively easier than assigning absolute scores because humans are better at relative judgments. It also directly measures what we care about: which model produces more preferred outputs. Additionally, pairwise comparisons naturally aggregate into preference rankings that can inform training through methods like RLHF, creating a direct connection between evaluation methodology and model improvement. The preference signal collected during evaluation is precisely the signal needed to train reward models in the RLHF pipeline.

The key insight is that pairwise comparisons produce ordinal information (A is better than B) rather than cardinal information (A scores 7.3, B scores 6.1). Ordinal information is sufficient for ranking models and for training reward models, which is why pairwise comparison has become the dominant paradigm in both evaluation and preference-based fine-tuning.

Alternative approaches use rating scales where evaluators score individual responses. This method offers different trade-offs: while potentially less reliable at the individual judgment level, it gives richer information about specific dimensions of quality:

Rating scales let more granular feedback and can identify specific weaknesses. For instance, a model might consistently score high on helpfulness but low on accuracy, revealing that it generates plausible-sounding but incorrect information. This diagnostic capability makes rating scales particularly useful during model development, where understanding the nature of failures is as important as measuring overall quality. However, they suffer from calibration issues: different evaluators may interpret "4 out of 5" differently, and even individual evaluators may shift their standards over the course of a long evaluation session.

The multi-dimensional nature of rating scales also reveals trade-offs that pairwise comparison can obscure. A response might score highly on helpfulness and coherence while scoring poorly on accuracy. A pairwise comparison would produce a single preference judgment that shows the evaluator's personal weighting of these dimensions. Rating scales make those dimensions explicit. This gives more actionable feedback for model improvement: you can see that Response A is better because it is more accurate, even though Response B is more clearly written.

A necessary concern in human evaluation is whether different evaluators agree. We measure this using inter-annotator agreement metrics. Agreement measurement matters because it tells us whether quality differences between responses are clear and objective, or whether subjective factors dominate the evaluation. High agreement suggests the evaluation is measuring something real and stable. Low agreement indicates either that the responses are similar in quality, that the evaluation criteria are ambiguous, or that the task itself admits multiple valid interpretations:

Cohen's Kappa accounts for agreement that would occur by chance. To understand why this correction matters, consider a three-way pairwise evaluation (A wins, B wins, Tie). If evaluators assigned labels randomly, they would agree 33% of the time simply by chance. A raw agreement of 60% sounds promising, but Kappa reveals it corresponds to only moderate agreement beyond what chance predicts. The formula for Cohen's Kappa is:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where:

• $p_o$ is the observed proportion of agreement between the two raters
• $p_e$ is the expected proportion of agreement if both raters labeled independently at random, computed as $\sum_k p_{k,1} \cdot p_{k,2}$ where $p_{k,r}$ is the proportion of category $k$ assigned by rater $r$
• $1 - p_e$ normalizes the maximum possible improvement over chance to 1, so $\kappa$ is bounded at 1

The key insight is that the numerator ($p_o - p_e$) measures how much better than chance the evaluators are performing, while the denominator ($1 - p_e$) tells us what the maximum possible improvement would be. Dividing one by the other puts agreement on a 0-to-1 scale where 0 means no improvement over chance and 1 means perfect agreement. Negative values are possible if raters agree less than chance would predict, which typically indicates systematic disagreement rather than random noise.

In the example above, two raters agreed on 80% of cases, and Kappa shows this is substantial agreement after adjusting for chance. This level of agreement is considered acceptable for instruction-following evaluation, though researchers often aim for Kappa above 0.7 before trusting results from a given evaluation protocol.

Human evaluation faces several practical challenges that limit its use as a primary evaluation mechanism during rapid development cycles.

Cost and scale create an immediate bottleneck. Evaluating thousands of examples across multiple models quickly becomes expensive. A single evaluation comparing two models on 1000 instructions with 3 annotators per comparison requires 3000 human judgments. At typical crowdsourcing rates, this can cost thousands of dollars and take days to complete. During rapid iteration cycles, this latency prevents teams from using human evaluation as a continuous feedback signal.

Evaluator expertise creates a quality problem for technical domains. For instructions requiring specialized knowledge (code debugging, mathematical derivations, scientific explanations), evaluators need domain expertise to judge correctness. A non-programmer might prefer a plausible-looking but buggy code response over a correct but terse one. Recruiting expert evaluators is expensive, and expert pools are often small enough that a few idiosyncratic raters can skew results.

Evaluation bias introduces systematic errors that may not be visible without careful analysis. Humans tend to prefer longer, more detailed responses even when brevity is appropriate. They may also favor responses that match their pre-existing beliefs, regardless of factual accuracy. These biases are particularly problematic because they are shared across evaluator populations, meaning they do not average out when you collect more judgments.

Cognitive load limits evaluation quality over time. Comparing long, complex responses is mentally taxing. Evaluator fatigue leads to increasingly inconsistent judgments during later evaluation sessions, creating temporal patterns in the data that contaminate results. Quality control mechanisms like attention checks and inter-rater agreement monitoring can detect but not eliminate this problem.

These challenges motivate the development of automatic evaluation methods that can scale while approximating human judgment. The goal is not to replace human evaluation entirely but to use it strategically: automatic metrics for rapid iteration, human evaluation for final validation and for calibrating automatic metrics.