Human Preference Data › Preference Collection Paradigms

Human preferences can be collected in several formats, each with different tradeoffs. Some formats scale easily but produce less reliable or less information-dense labels. The format we choose shapes the annotation process and the mathematical models we can apply to learn from the data. We must consider both the cognitive demands on annotators and the statistical properties of the resulting data. A format that is cognitively demanding produces fewer annotations per hour and shows more noise; a format that is statistically inefficient requires more data to learn the same signal. Good preference collection design balances these tradeoffs.

The central challenge is that quality is a multidimensional concept. A response can be helpful but inaccurate, concise but incomplete, appropriately safe but overly cautious. Different collection formats make different choices about how to handle this multidimensionality. Some collapse it into a single comparison; others try to capture it explicitly through structured dimensions. Understanding these tradeoffs helps you choose the right format for your alignment goals and design collection workflows that minimize systematic distortions.

The dominant paradigm in RLHF presents annotators with two model responses to the same prompt and asks which is better. This format has become the standard for preference collection because it addresses both human and mathematical needs simultaneously.

From the annotator's perspective, pairwise comparisons reduce a complex quality judgment to its simplest possible form. Instead of asking "how good is this response?" we ask "which of these two responses is better?" This reframing has several advantages that compound to make the data much more reliable.

First, consider cognitive simplicity. Comparing two options is much easier than rating on an absolute scale. The annotator need only decide which response they would rather receive, not quantify exactly how much better one is than the other. Humans are generally better at making relative judgments than absolute ones. Think of asking a wine critic to rate a wine from 1 to 100 versus asking them which of two wines they prefer: the comparative task is less ambiguous and less subject to individual scale interpretation.

Second, pairwise comparisons are calibration-free. Annotators don't need to agree on what "4 out of 5" means. When two annotators both prefer response A over response B, they have communicated the same information regardless of whether they would have assigned different numerical ratings. This removes a significant source of noise in absolute rating approaches, where annotators with different internal scales produce incomparable scores even when their underlying preferences are identical.

Third, pairwise data aligns naturally with the Bradley-Terry model we'll cover in the next chapter. This mathematical framework was specifically designed to learn underlying quality scores from pairwise comparisons, and it provides theoretically grounded aggregation of preferences across annotators and comparisons. The data format and the mathematical model are matched to each other.

A typical pairwise comparison looks like:

The annotator selects which response better satisfies the implicit quality criteria for the given prompt and audience. Most preference datasets discard or minimize tie votes, as they provide less signal for training reward models. This reflects a main point: the most valuable information comes from cases where annotators can make a clear distinction between options. Ties represent uncertainty in the signal, and while they contain some information (the annotator could not distinguish between the responses), they are harder to use effectively in training.

Some annotation schemes offer a more granular response set, such as "A is much better," "A is slightly better," "Tie," "B is slightly better," "B is much better." This captures strength of preference, not just direction, and can provide richer training signal. The tradeoff is added cognitive complexity: annotators must calibrate what "slightly" versus "much" means, reintroducing some of the calibration problems that pairwise comparison was designed to avoid.

An alternative approach asks annotators to rate individual responses on a numerical scale (e.g., 1-5 or 1-7). This captures absolute quality judgments:

Likert ratings provide more information per annotation, but they suffer from calibration issues that can be severe in practice. Different annotators interpret the scale differently, and even the same annotator may shift their standards over time. One annotator's "4" might correspond to another annotator's "3," making it difficult to aggregate ratings across annotators without careful normalization procedures.

The calibration problem also interacts with domain: annotators may have different standards for different types of tasks. An annotator with deep coding expertise might rate a mediocre code explanation as "2," while a non-programmer rates the same response as "4." These differences reflect different standards rather than annotator error, but they are difficult to disentangle from quality differences.

Despite these challenges, Likert ratings have one significant advantage: they allow you to evaluate a response without needing another response to compare it against. This matters when you have a large pool of responses and cannot enumerate all possible pairs. Ratings can be converted to implied preferences, but the conversion requires assumptions about scale interpretation that introduce additional noise.

Some collection schemes present more than two responses and ask for a full ranking:

Rankings generate multiple pairwise comparisons from a single annotation task. Specifically, ranking $k$ items yields $\binom{k}{2} = k(k-1)/2$ pairwise comparisons, since we learn the relative ordering of every pair. For $k = 4$ items, this yields 6 pairwise comparisons; for $k = 8$ items, 28 comparisons. In theory, this is highly efficient.

In practice, however, the efficiency gain is smaller than it appears. Ranking becomes cognitively demanding as the number of items increases. Annotators may carefully compare the top two or three responses but make increasingly arbitrary decisions among lower-ranked items. This produces high-quality signal for the top comparisons and noisy signal for the rest. Studies of annotation quality in ranking tasks confirm this pattern: inter-annotator agreement drops substantially for the lower-ranked comparisons.

There is also a practical concern about ranking consistency. Ranking requires the annotator to hold multiple items in working memory simultaneously and make transitively consistent judgments. Human judgment is not always transitive: an annotator might prefer A to B, B to C, but prefer C to A when compared directly. Rankings force transitivity by construction, which can introduce distortions when the annotator's actual preferences are cyclic or context-dependent.

A newer approach, developed largely by Anthropic, asks annotators to evaluate responses against an explicit set of principles or a "constitution." Rather than just asking "which is better?" the interface presents the constitution and asks "which response better follows these principles?"

This approach has the advantage of making the evaluation criteria explicit and consistent. Annotators are not relying on personal intuition about what "better" means; they are applying a shared framework. The disadvantage is that the constitution must be carefully designed, and annotators must understand and apply it correctly. If the constitution is ambiguous, different annotators will interpret it differently, reintroducing the calibration problem.

Constitutional evaluation is also useful for multi-dimensional quality assessment. A constitution might specify separate criteria for helpfulness, harmlessness, and honesty, allowing annotators to mark which criterion a response fails. This produces richer signal than a single preference judgment, though it requires more annotation time per comparison.

Despite the information efficiency of rankings and the richness of ratings, pairwise comparisons remain the standard for several reasons that reinforce each other in practice.

The first reason is reliability. Binary choices show higher inter-annotator agreement than multi-point scales. When forced to make a simple choice between two options, annotators tend to agree with each other more consistently than when asked to assign numerical scores or produce full rankings. This higher agreement translates directly into cleaner training signal for reward models. If two annotators looking at the same pair both prefer Response A, we have strong evidence that Response A is better. If they assign different ratings (say, 3 and 5), we have much weaker signal.

The second reason is theoretical grounding. The Bradley-Terry model provides a principled way to aggregate pairwise comparisons into reward scores. This model assumes that each response has some underlying "quality" score and that the probability of preferring one response over another follows a logistic function of the difference in their quality scores. We will explore this model in depth in the next chapter, but the key point is that pairwise data fits naturally into this framework, enabling principled learning from diverse comparison data.

The third reason is practical simplicity. The interface is straightforward to build, the data format is simple to process, and the annotation task is easy to explain to new annotators. These practical considerations matter enormously when scaling to thousands of comparisons across many annotators. Simpler collection pipelines have fewer points of failure and are easier to quality-control.

Most major preference datasets, including those used to train InstructGPT, Claude, and Llama 2, rely primarily on pairwise comparisons, which validates this choice empirically across multiple organizations and use cases.