Human Preference Data › Comparison Design

The prompts and responses used in preference collection determine what behaviors the resulting model will learn. Poor comparison design limits what alignment can achieve, no matter how good your annotators are. This section explores how to construct comparisons that provide maximal information about the quality distinctions we care about.

The central principle of comparison design is that you get the model behavior you measure for. If your preference dataset consists entirely of factual questions about science, your aligned model will learn to produce good science answers but may not generalize to other domains. If your comparisons only contrast clearly good responses against clearly bad ones, your model will learn to avoid obvious failures but won't learn to make fine-grained quality distinctions in the middle of the quality distribution. Intentional, principled comparison design produces alignment that generalizes in the directions you intend.

The prompts used for preference collection should cover the distribution of tasks the model will encounter in deployment. This requires careful consideration of multiple dimensions that together determine the coverage and informativeness of your dataset.

Domain coverage is the most obvious consideration. Instructions should span different use cases. Creative writing and coding assistance require different behavior from factual questions or advice requests. Analysis and summarization add further demands, as does translation. A dataset heavily weighted toward one domain will produce a model with uneven capabilities, excellent in the covered domain but poorly calibrated in others. The distribution of prompts in your preference dataset should match, as closely as possible, the distribution of requests the model will receive in production.

Difficulty distribution is equally important but often neglected. Including both simple and complex prompts ensures the model learns preferences across difficulty levels. Easy prompts, like basic factual questions with clear answers, may have nearly universal agreement among annotators and provide clear but shallow signal. Harder prompts, like subtle ethical dilemmas or ambiguous requests, generate more disagreement but provide richer signal about what good judgment looks like in difficult cases. A good preference dataset includes prompts across the full difficulty spectrum.

Edge cases deserve deliberate inclusion. Prompts that probe potential failure modes should be specifically added to the dataset: ambiguous requests where the intended meaning is unclear, requests with hidden assumptions that may not hold, requests that might elicit harmful content if handled poorly, and requests where the appropriate response involves acknowledging uncertainty or declining. These adversarial prompts help the model learn appropriate boundaries and exception handling.

Prompt diversity can be achieved through several strategies. Organic collection gathers prompts from actual user interactions with early model versions, capturing the true distribution of real usage. Templated generation creates prompts by filling templates with varied content. This keeps systematic coverage of topic and difficulty combinations. Adversarial generation deliberately crafts prompts that probe edge cases, using techniques like red-teaming to find prompts that expose model weaknesses.

The responses being compared typically come from the model being aligned, or an earlier version of it, but the generation strategy affects what the model can learn. Different generation strategies expose different aspects of the model's behavior distribution.

Temperature diversity is a standard technique. Generating responses at different temperatures produces variety in style and quality. High-temperature samples (temperature above 1.0) produce creative but sometimes risky or incoherent outputs. Low-temperature samples (temperature near 0) produce safe but generic outputs that may not show useful variation. Comparing responses generated at different temperatures helps the model learn when creativity versus caution is appropriate for different types of prompts.

Using multiple model versions or models is another common strategy. Comparing outputs from the current model against a previous version creates informative contrasts that help the model learn from its own past mistakes. Comparing against a different model entirely, such as using GPT-4 outputs to create positive examples for training a smaller model, creates strong directional signal about what ideal outputs look like. If annotators consistently prefer GPT-4's responses over the model being trained, those comparisons teach the model to emulate GPT-4's style and quality.

Intentional degradation involves deliberately including low-quality responses to establish clear negative examples. These might be truncated responses, factually wrong responses, stylistically poor responses, or responses that fail to follow instructions. Easy comparisons where one response is clearly much better than the other provide strong, low-noise signal about basic quality expectations. However, these easy comparisons are less informative than close comparisons between two reasonable responses, because they don't teach the model to make fine-grained distinctions. A good dataset includes both easy and hard comparisons.

The most informative comparisons involve responses that differ in specific, meaningful ways. Rather than comparing a clearly good response to a clearly bad one, effective datasets include pairs where both responses are reasonable but differ along dimensions that matter for quality. These contrastive pairs force annotators, and later the reward model, to make fine-grained distinctions.

Effective contrastive pairs include situations where:

• Both responses are reasonable but differ in style or emphasis, teaching the model which style is preferred for different contexts.
• One response is more complete while the other is more concise, teaching the model to calibrate length to the complexity and nature of the request.
• One response directly answers the question while the other provides useful context first, teaching the model appropriate ordering of information.
• Both responses are correct but one is more clearly explained, teaching the model the value of clarity over technical correctness alone.
• One response acknowledges uncertainty while the other states the same information confidently, teaching the model appropriate epistemic calibration.

The key insight is that we learn most from close comparisons where the distinction is subtle, not from obvious cases where any reasonable annotator would agree. A dataset composed entirely of easy comparisons produces a reward model that can identify poor responses but cannot distinguish between good and excellent ones, which is often the more important distinction in practice.