Human Preference Data › Summary

Human preference data provides the empirical foundation for language model alignment. Rather than specifying desired behavior through rules, we collect examples of humans choosing between model outputs, then use these choices to train reward models that guide model behavior. The simplicity of this idea belies the substantial engineering and judgment required to execute it well.

The key elements of preference data collection include:

• Collection format: Pairwise comparisons dominate due to cognitive simplicity and theoretical grounding with the Bradley-Terry model, though rankings and ratings offer alternatives with different tradeoffs in information density and annotator cognitive load.
• Interface design: Randomized response positions, consistent formatting, hidden model identity, and forced attention mechanisms prevent systematic biases from corrupting the data. Small interface decisions compound into large dataset-level effects at scale.
• Comparison design: Prompts should cover expected use cases across domains and difficulty levels, including edge cases that probe failure modes. Response pairs should include both easy comparisons that anchor the quality scale and hard, contrastive comparisons that teach fine-grained distinctions.
• Annotator guidelines: Clear, specific criteria for quality dimensions (helpfulness, accuracy, harmlessness, honesty, conciseness) with explicit priority orderings for conflicts and worked calibration examples help ensure consistent judgments across annotators and over time.
• Quality control: Inter-annotator agreement metrics (Cohen's Kappa, Krippendorff's Alpha), gold questions, consistency checks, annotation time tracking, and statistical outlier detection together maintain data reliability through the collection process. No single mechanism is sufficient; layered defenses are necessary.
• Bias analysis: Length bias, position bias, and demographic bias in the annotator pool must be actively monitored and mitigated throughout collection, not just at the end.

The decisions made during preference collection cascade through the entire alignment pipeline. In the next chapter, we'll see how the Bradley-Terry model provides a principled framework for converting pairwise preference data into the reward scores that drive RLHF training.