Human Preference Data › Real-World Preference Datasets

Several public preference datasets demonstrate these principles at scale, and studying them reveals the design choices that different organizations have made and their consequences.

Anthropic HH-RLHF contains approximately 169,000 conversations rated for helpfulness and harmlessness. Annotators compared assistant responses in multi-turn dialogues, focusing on whether responses were helpful without being harmful. The dataset explicitly separates the helpfulness and harmlessness dimensions. This provides separate labeled subsets for each. This design choice reflects the tension between helpfulness and safety: responses that are maximally helpful may sometimes be harmful, and perfectly safe responses may refuse to be helpful. Studying where this tension appears is itself informative.

OpenAssistant is a crowdsourced dataset with multi-level rankings. Annotators ranked multiple responses per prompt, and the data includes quality flags, annotator metadata, and multi-turn conversation structure. The crowdsourced nature provides diverse annotator perspectives but introduces more variability in annotation quality than professionally managed annotation projects. OpenAssistant demonstrates both the potential and the challenges of large-scale community annotation.

Stanford Human Preferences (SHP) is derived from Reddit, where upvotes serve as implicit preference signals. When a user's comment receives more upvotes than another comment on the same post, we can interpret this as a collective preference for the higher-upvoted response. This represents a fundamentally different collection paradigm: instead of explicit annotation by paid workers, we harvest naturally occurring human feedback from existing social data. The advantages are scale (millions of comparisons) and authenticity (real human judgments in natural context). The disadvantages include noisiness (upvotes reflect many factors beyond response quality), demographic skew (Reddit users are not representative of all users), and the inability to control what types of comparisons are available.

UltraFeedback uses GPT-4 to provide preference judgments, representing the AI feedback approach where a stronger model labels data for training weaker models. This is sometimes called RLAIF (Reinforcement Learning from AI Feedback) rather than RLHF. The approach is far more scalable than human annotation: GPT-4 can evaluate thousands of comparisons per hour at low cost. The quality of the resulting signal depends on the quality of the judge model, and GPT-4 has its own biases and blind spots that transfer into the training data. Research has shown that RLAIF can produce competitive results to RLHF in many settings, but the two approaches may have different failure modes.

Each dataset makes different design choices that affect what behaviors the resulting models learn. Understanding these choices helps practitioners select appropriate data for their alignment goals and diagnose why models trained on different datasets exhibit different behavioral patterns.