Human Preference Data › Annotator Guidelines

Clear guidelines are essential for consistent preference judgments. Without explicit criteria, different annotators will apply different standards, introducing noise into the dataset that cannot be easily distinguished from legitimate disagreement about quality. Think of annotator guidelines as the shared vocabulary of quality: they ensure that when two annotators say "this response is better," they mean something similar by "better."

Guidelines serve multiple functions simultaneously. They communicate what quality dimensions matter and how to weight them. They provide examples that calibrate annotators' internal standards to a common reference point. They address ambiguous cases before annotators encounter them, preventing inconsistent ad-hoc decisions. And they create a feedback loop: as annotation proceeds, new edge cases emerge that can be documented and added to the guidelines, progressively improving consistency over time.

The investment in guidelines pays off disproportionately. A poorly specified guideline leads to systematic biases that corrupt thousands of annotations before they are detected. A well-specified guideline, by contrast, reduces noise across the entire dataset and makes downstream reward model training more efficient by providing cleaner signal.

Guidelines should specify what annotators should consider when comparing responses. These criteria operationalize our abstract notion of "quality" into concrete factors that annotators can evaluate independently. Good criteria are specific enough to guide consistent judgment but not so mechanical that they prevent an overall evaluation.

Common criteria in RLHF guidelines include:

• Helpfulness: Does the response help you accomplish your goal? Does it answer the question asked, or does it dodge the question with tangentially related information? Does it provide actionable guidance rather than vague generalities? A response that sounds impressive but doesn't help the user accomplish their task fails on helpfulness even if it is technically accurate.
• Accuracy: Is the information correct? For factual claims, does the evidence support the assertion? For reasoning tasks, are the logical steps valid and is the conclusion supported by the premises? For code, does the code work as described? Accuracy is often the highest-priority criterion because inaccurate information actively harms users.
• Clarity: Is the response easy to understand? Is it well-organized? Does it explain technical concepts at an appropriate level for the apparent audience? A response that is technically correct but incomprehensible to the intended reader fails on clarity. Good clarity means the reader spends cognitive effort understanding the topic, not deciphering the response.
• Harmlessness: Does the response avoid providing dangerous or harmful information? Does it refuse inappropriate requests appropriately without being unnecessarily restrictive? A response that correctly refuses to help with a clearly harmful request should be preferred over one that complies. But a response that refuses a legitimate request out of excessive caution is also penalized on this dimension.
• Honesty: Does the response acknowledge uncertainty when appropriate? Does it avoid claiming capabilities the model doesn't have? Does it represent the state of knowledge accurately rather than fabricating confident-sounding answers? Epistemic honesty is particularly important for preventing harmful overconfidence.
• Conciseness: Does the response communicate what it needs to without unnecessary padding? Is it as long as necessary but no longer? This criterion explicitly counteracts length bias by making conciseness a quality dimension in its own right.

Guidelines should specify how to weigh these criteria when they conflict. For example: "If one response is more helpful but slightly less accurate, prefer the accurate response. Accuracy takes priority over helpfulness." These explicit priority rules reduce the arbitrariness of difficult comparisons and help ensure that different annotators resolve the same conflict the same way.

The criteria themselves should also be domain-adapted where appropriate. The most important quality dimensions for a coding question differ from those for a creative writing request. Some annotation systems use different guideline versions for different task types. This keeps the quality framework is always appropriate for the type of comparison being made.

Some comparisons have no clear winner, and guidelines must address this directly. Leaving ambiguous cases unspecified leads annotators to develop their own inconsistent resolution strategies, introducing noise that is particularly harmful because it concentrates in exactly the hard cases where clear signal is most valuable.

For ties, guidelines should specify when annotators may select "tie" or "no preference" and when they should push through to a decision. Some annotation projects discourage ties almost entirely, on the grounds that even a small preference difference provides training signal, while forcing annotators to decide pushes them to examine the responses more carefully. Others allow ties freely, preferring not to force arbitrary choices that add noise. The guideline should be explicit about which policy applies, and the policy should be chosen based on how ties will be handled in downstream training.

For cases where responses take different valid approaches, guidelines should specify resolution strategies. Two responses might both correctly answer a programming question but use different algorithms, or both correctly explain a concept but use different analogies. If both responses would fully satisfy a reasonable user, ties may be appropriate. If one approach is more pedagogically effective or more efficient in practice, annotators should prefer it and the guidelines should specify how to recognize that.

For cases with partial quality differences, where one response is better on some criteria and worse on others, guidelines should provide an explicit priority ordering or aggregation rule. Without this, annotators will apply their personal intuition about the relative importance of different criteria, introducing systematic variation based on annotator characteristics rather than response quality.

Including worked examples in the guidelines is one of the most effective investments in annotation quality. Calibration examples transform abstract criteria into concrete demonstrations that anchor annotators' internal quality standards. They show rather than tell, which is particularly important for subtle quality distinctions that resist pure verbal description.

Each calibration example should show the full context: the prompt, both responses, the correct choice, and a detailed explanation of why that choice is correct. The explanation should explicitly reference the quality criteria in the guidelines. This shows how to apply them. This creates a direct bridge between the abstract criteria and the concrete practice of annotation.

Effective calibration examples serve specific pedagogical functions:

• Clear cases establish the baseline: an obviously good response versus an obviously poor one. This shows what the extremes of the quality scale look like and making the criteria feel concrete.
• Close calls illustrate subtle distinctions: two good responses where one is slightly better on a specific dimension. This shows how to make fine-grained judgments when both options are reasonable.
• Traps catch common mistakes: cases where surface features like length, formality, or confidence might mislead annotators away from the better response. For example, a confident-sounding but factually wrong response compared to a hesitant but accurate one, where the guidelines should make clear that accuracy takes priority over apparent confidence.
• Edge cases document hard problems: ambiguities that the guidelines resolve in a specific way, even if the resolution is somewhat arbitrary, so that different annotators handle the same type of case consistently.

The calibration examples should be expanded over time as new patterns emerge. Every time a project manager reviews a batch of annotations and notices systematic inconsistency, the team should diagnose and document it. They can then add a calibration example or clarify the existing guidelines.

As annotation proceeds, edge cases emerge that the original guidelines didn't anticipate. Maintaining a living FAQ or edge case document helps ensure consistent handling across annotators and over time. This document should be version-controlled and accessible to all annotators, with new entries added as decisions are made.

The act of documenting edge cases as they arise forces explicit decisions that improve consistency. Without documentation, different annotators will make different decisions for the same case type, introducing systematic variation that looks like random noise but reflects policy inconsistency.