Human Evaluation Design › Cost Estimation and Budgeting

Human evaluation is expensive, but poor evaluation is more expensive. Underestimating costs leads to underpowered studies, rushed collection, and ultimately, misleading conclusions about model performance. When you cannot afford to collect enough data to detect real differences, you risk concluding that two models perform equally when one is superior, or you may overfit to small samples that don't stand for true performance differences. Accurate budgeting requires breaking down the components of annotation cost and understanding how they scale with your study design.

Break down annotation time into components. Accurate time estimation requires understanding the cognitive operations involved in your specific task and how long each takes on average.

Reading time: The time to consume the input and output. For a 500-word document and 100-word summary at 200 words per minute: $(500 + 100) / 200 = 3$ minutes. Note that reading speed varies by content difficulty; technical documents require more time than simple narratives. You should adjust words-per-minute estimates based on the complexity of your domain.

Judgment time: The time to apply criteria and render a verdict. Simple Likert ratings take 15-30 seconds. Complex error annotation takes 2-5 minutes. Judgment time increases with the number of criteria annotators must consider and the difficulty of the comparison. Comparing two long responses for subtle differences takes longer than rating a single response on a clear objective criterion.

Interface overhead: Loading times, reading instructions, and submitting responses. Budget 10-15 seconds per item. While this seems small, it adds up over thousands of items. Interface friction, such as slow loading or complex navigation, increases overhead and reduces effective hourly output.

For a summarization task with 1,000 examples:

• Reading: 3 minutes $\times$ 1,000 = 3,000 minutes
• Judgment: 0.5 minutes $\times$ 1,000 = 500 minutes
• Overhead: 0.25 minutes $\times$ 1,000 = 250 minutes
• Total: 3,750 minutes (62.5 hours)

With 3 annotators per item for reliability (see the next chapter on Inter-Annotator Agreement), you need 187.5 hours of annotation time. This replication factor is needed for measuring reliability but triples your labor costs compared to single annotation.

It is worth pilot-testing your time estimates before committing to a budget. Ask three or four people to annotate ten items each and time them. Real annotation times consistently exceed initial estimates, often by 30-50%, because researchers underestimate the time annotators spend rereading unclear passages, reconsidering initial judgments, and working through interface details. A pilot-derived time estimate is worth more than any formula.

Annotation costs depend on the number of items, how many annotators review each item, the time spent per item, and associated fees. To calculate the total cost, we multiply the number of annotation instances by the cost per instance, which includes both labor and fixed costs:

$$\text{Total Cost} = N \times R \times (T \times W + F)$$

where:

• $N$: number of items to annotate
• $R$: number of annotators per item (replication factor)
• $T$: time required per item (in hours)
• $W$: hourly wage (in dollars)
• $F$: fixed platform fees per task (in dollars)

This formula captures the linear scaling of costs with dataset size ($N$) and reliability requirements ($R$), while separating variable labor costs ($T \times W$) from per-task overhead ($F$). The fixed fee $F$ covers platform infrastructure, payment processing, and service charges that don't scale with task duration.

For our summarization example with US annotators ($18/hour) and 15% platform fees:

• $N = 1{,}000$, $R = 3$, $T = 0.0625$ hours, W = \18$,$F = $0.10$

$$\begin{aligned} \text{Total Cost} &= 1000 \times 3 \times (0.0625 \times 18 + 0.10) \\ &= 1000 \times 3 \times (1.125 + 0.10) \\ &= 1000 \times 3 \times 1.225 \\ &= \$3{,}675 \end{aligned}$$

This formula looks straightforward, but several hidden costs can inflate the actual budget materially. Rejected and re-annotated items add 5-15% to costs in most studies. Annotator dropout mid-task requires recruiting replacements. Data cleaning and quality control takes researcher time that should be budgeted even if it doesn't appear in the annotation platform's invoice. A realistic budget adds 25-40% to the formula estimate as a contingency.

International annotators reduce costs but may introduce cultural bias. Consider stratified sampling: 70% US/UK annotators for linguistic nuance, 30% international for global perspective, at blended rates. This approach balances cost savings with the need for native speaker intuition on subtle linguistic phenomena.

Rushing annotation to save money often backfires. When annotators face time pressure, they optimize for speed rather than accuracy, leading to systematic biases that contaminate your data. Annotators working under tight time pressure show:

• Anchoring bias: Rating subsequent items similarly to previous ones to reduce cognitive effort
• Satisficing: Selecting the first acceptable option rather than the best one
• Attrition: Quitting the task, leaving you with incomplete data from the most difficult items

Budget for 20-30% longer than your time estimate. This buffer allows annotators to maintain quality and gives time for the clarifying questions that inevitably arise. Annotators who feel they have sufficient time to do good work produce more consistent judgments and are more likely to continue working on your tasks in the future, reducing recruitment costs for subsequent studies.

There is an important distinction between paying a fair hourly wage versus creating time pressure through piece-rate pricing. Piece-rate pay (paying per item rather than per hour) creates strong incentives for speed, which is useful when you want volume but harmful when you need accuracy. For quality-necessary evaluations, hourly or task-based compensation that does not penalize slower, more careful workers produces better data than per-item compensation that rewards speed.

You need enough judgments to detect real differences between models. The required sample size depends on several factors:

• Effect size: How large a quality difference do you care about? Detecting a 0.1-point difference on a 5-point scale requires more data than detecting a 0.5-point difference. Small effect sizes may be statistically significant but practically meaningless, while large effect sizes are both significant and consequential.
• Variance: How much do annotators disagree? High-variance tasks (creative writing evaluation) need more samples than low-variance tasks (grammaticality judgment). You can estimate variance from pilot studies or similar published work.
• Alpha level: Your tolerance for false positives (typically 0.05).

Use power analysis to determine minimum sample sizes. For comparing two systems with paired t-tests, expecting a medium effect size (Cohen's $d = 0.5$) with 80% power and $\alpha = 0.05$, you need approximately 34 paired judgments. But this assumes clean data; with noisy annotations, you may need 100-200 per condition. Power analysis protects you from false negatives. This keeps if a real difference exists, your study is able to detecting it.

One important nuance: power calculations assume you know the expected effect size and variance before collecting data, which is rarely true. This is why pilot studies are so useful. A 50-item pilot gives you empirical estimates of both quantities, letting much more accurate power calculations than theoretical guesses. The cost of a pilot is usually small relative to the full study and can save you from the far greater cost of running an underpowered study that fails to detect a real difference.