Human Evaluation Design › Worked Example: Designing a Helpfulness Evaluation

Let's design a complete evaluation for assessing the helpfulness of chatbot responses. This example illustrates how the elements we've discussed, including interface design, instructions, annotator selection, and cost estimation, integrate into a cohesive study. By walking through each decision point, we show how theoretical principles translate into practical implementation.

We want to evaluate whether a conversational AI gives helpful responses to user queries. "Helpfulness" is multidimensional: it includes relevance, accuracy, completeness, and tone. Rather than treating helpfulness as a monolithic concept, we recognize that a response can be accurate but incomplete, or complete but irrelevant. Our evaluation must capture these tradeoffs or explicitly prioritize certain dimensions over others.

For this study, we choose to evaluate overall helpfulness using a preference paradigm rather than decomposing into subdimensions. The motivation is that we want to compare two model versions (a baseline and a new candidate) and determine which is preferable to users, not to diagnose specific strengths and weaknesses. A preference paradigm is faster to annotate and more reliable for this comparison purpose than multidimensional absolute ratings.

We choose a side-by-side comparison interface rather than absolute ratings because we want to compare two model versions (baseline vs. new). The interface displays:

• The user query at the top (sticky header)
• Response A on the left, Response B on the right (randomized order)
• Three radio buttons below: "A is better," "B is better," "Tie"
• Optional text boxes for explaining the choice (shown only if "A is better" or "B is better" is selected)

This design minimizes scrolling and keeps the decision criteria visible. The sticky header ensures annotators always have the user query in view while comparing responses. Randomization of left/right positioning controls for order effects, while the optional explanation field allows us to collect qualitative data on why annotators prefer certain responses without forcing them to write text for every comparison.

We add a brief checklist below the radio buttons that reminds annotators of the key criteria without requiring them to leave the current item. This "criteria cheat sheet" reduces the working memory load of holding criteria in mind while reading two sometimes lengthy responses.

Our instructions include:

Task Overview: "You will see a conversation between a user and two different AI assistants. Read the user's question, then decide which assistant provided the more helpful response."

Criteria Definition:

• A helpful response directly addresses the user's question
• A helpful response is factually accurate (if making factual claims)
• A helpful response is appropriate in tone (not overly formal nor casually inappropriate)
• If both responses are equally helpful (or unhelpful), select "Tie"

These criteria give annotators multiple dimensions to consider while giving a clear tie-breaking rule. By specifying that factual accuracy only matters "if making factual claims," we prevent annotators from penalizing responses to subjective questions for lacking citations.

Examples:

• Positive: User asks for a Python sorting algorithm. Response A gives working code with comments; Response B suggests using Google. A is better.
• Edge case: User asks for medical advice. Both responses refuse to answer (correctly). Select "Tie" unless one refusal is materially more informative about why it cannot answer.

The positive example establishes the pattern of comparing two specific responses, while the edge case prevents annotators from penalizing appropriate refusals. Medical advice refusals stand for a common failure mode where annotators might penalize safety measures as unhelpfulness; the example explicitly validates safety-conscious behavior.

We require:

• Native English speakers (verified via qualification test)
• Age 18+ (diversity across age groups)
• Mix of technical and non-technical backgrounds (to evaluate both coding and general queries)

We create a qualification test with 10 examples covering obvious cases (one response is clearly wrong) and subtle cases (both are adequate but one is more thorough). Annotators must score 8/10 to qualify. The technical/non-technical split ensures our evaluation doesn't optimize solely for software engineers while remaining capable of judging technical content. Age diversity captures generational differences in communication preferences and expectations of AI systems.

We also add a brief demographic survey at the end of the qualification phase (not gatekeeping, just informational) to record annotators' backgrounds and use that information when diagnosing disagreement patterns later.

Parameters:

• 500 conversations to evaluate
• 3 annotators per conversation (to measure agreement)
• Estimated time per judgment: 1 minute (reading query: 15s, reading both responses: 30s, decision: 15s)
• Wage: $20/hour (premium for English native speakers)
• Platform fee: 20%

Calculation:

$$\begin{aligned} \text{Total Cost} &= 500 \times 3 \times \left(\frac{1}{60} \times 20 \times 1.20\right) \\ &= 500 \times 3 \times 0.40 \\ &= \$600 \end{aligned}$$

We budget $750 to account for quality checks, re-annotations of disputed items, and pilot testing. The buffer covers approximately 25% additional cost. This gives flexibility for the inevitable complications that arise in real data collection.