Human Preference Data › Human Preference Data

The alignment problem, as we discussed in the previous chapter, fundamentally comes down to a question of measurement: how do we capture what humans want from language models? Abstract concepts like helpfulness, harmlessness, and honesty resist simple formalization. You cannot write a loss function that directly optimizes for "be helpful without being sycophantic." The space of possible model behaviors is vast, the space of desirable behaviors is hard to characterize, and models tend to fail in the gap between what a rule captures and the intended behavior.

Think of trying to specify "good teaching" through a rubric. You might write rules like "explains concepts clearly," "uses examples," "checks for understanding," and "adapts to the student's level." But any experienced teacher knows these rules capture only a fraction of what makes teaching effective. A teacher who follows all the rules mechanically but fails to engage students or respond intuitively to their needs still falls short in ways the rubric doesn't measure. The same problem plagues rule-based alignment: the rules capture the surface structure of good behavior while missing the deeper patterns that matter.

Human preference data is the empirical foundation for alignment precisely because it sidesteps this specification problem. Rather than trying to formalize desired behavior through rules or heuristics, we collect examples of humans choosing between model outputs. These choices encode implicit knowledge about what makes a response good: whether it answers the question correctly, whether the tone is appropriate, whether it avoids harmful content, whether it acknowledges uncertainty when uncertain. The approach can capture subtle preferences that resist explicit specification. When a human annotator chooses Response A over Response B, they are expressing something richer than any rule could capture: an integrated judgment across multiple quality dimensions.

The main point is that preference data turns an ill-specified optimization problem into a well-specified one. Instead of optimizing "be helpful," we optimize "produce responses that humans prefer," and we have direct empirical measurements of those preferences. This reframing is powerful even though it introduces new challenges: preferences vary across humans, reflect biases and context, and may not always capture what we want in the long run.

This chapter covers the practical machinery of preference data collection in depth: how to design comparison interfaces, what makes a good preference dataset, how to write annotator guidelines that produce consistent judgments, how to measure and maintain data quality, and how to detect and correct systematic biases. These decisions affect the entire alignment pipeline, shaping the reward models we will train and ultimately determining how our models behave in deployment. The quality of your preference data is the ceiling on the quality of your alignment.