Human Evaluation Design › Key Parameters
Language AI HandbookAll chapters Human Evaluation Design › Key ParametersOne section · Sections in this chapter9 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for evaluation cost calculation are:
-
n_items: Number of items to annotate. Scales linearly with total cost. This is the number of examples, conversations, or outputs you need evaluated to reach your statistical power requirements.
-
n_annotators_per_item: Number of annotators per item (replication factor). Higher values improve reliability through aggregation but increase cost linearly. This parameter directly impacts your ability to measure inter-annotator agreement and the stability of your quality estimates.
-
minutes_per_item: Estimated time required to annotate a single item. Depends on task complexity, reading time, and judgment difficulty. Complex tasks like error annotation require more time than simple preference judgments.
-
hourly_wage: Annotator hourly wage in dollars. Varies by geography, expertise level, and platform. US workers typically command $15-25/hour while international workers may accept lower rates, though potentially with tradeoffs in cultural familiarity.
-
platform_fee_percent: Platform fee as a decimal fraction (e.g., 0.20 for 20%). Includes payment processing and service fees. Most commercial platforms charge between 15% and 30% on top of worker wages.
For statistical power analysis:
-
effect_size: Expected standardized difference between systems (Cohen's ), where 0.2 is small, 0.5 is medium, and 0.8 is large. Smaller effects require substantially larger samples to detect. Effect sizes below 0.3 are often difficult to detect reliably in human evaluation due to annotator noise.
-
alpha: Significance level for hypothesis testing (typically 0.05), representing the probability of false positives. Lower alpha values (e.g., 0.01) require larger samples but give stronger evidence against the null hypothesis.
-
power: Desired statistical power (typically 0.80), representing the probability of correctly detecting a true effect. Power below 0.80 risks missing real improvements, while power above 0.90 often requires prohibitive sample sizes.
For reliability simulation:
- annotator_noise: Standard deviation of random error in annotator judgments. Lower values indicate more consistent annotators and higher inter-annotator agreement. Typical values range from 0.1 (high agreement) to 0.3 (noisy judgments).