RLAIF › AI Preference Generation

With constitutional principles in place, the next step is generating preference data at scale. This process requires careful prompt engineering and consideration of potential failure modes. The goal is to produce preference labels that reliably capture the quality distinctions encoded in the constitution, while minimizing noise and systematic biases that could corrupt the training signal. Think of this as the quality control layer of the RLAIF pipeline: the constitution defines what good responses look like, but the preference generation mechanism determines how reliably those definitions translate into useful training labels.

The prompt structure materially affects preference quality. The way we frame the evaluation task influences how the AI interprets its role and applies the constitutional principles. Several strategies improve AI preference reliability, and understanding why each works helps you adapt them to new situations.

Pairwise comparison: Present both responses simultaneously and ask which is better. This mirrors human annotation protocols and allows direct comparison. The simultaneous presentation is important because it lets the evaluator to make relative judgments, comparing specific features of each response rather than trying to assess absolute quality. Relative judgments tend to be more reliable because they require less calibration. When you ask "which is better," you are asking for a comparison that can succeed even if the evaluator's absolute quality scale is poorly calibrated.

Chain-of-thought evaluation: Asking the AI to explain its reasoning before stating a preference often produces more reliable judgments, mirroring the benefits of chain-of-thought reasoning we discussed in earlier chapters. When the model must articulate why one response is better, it engages more deeply with the evaluation criteria and is less likely to make superficial judgments based on surface-level features like response length. The reasoning trace also gives useful information for debugging and improving the constitution: if the model's explanations reveal that it is applying principles in unexpected ways, that is a signal to revise those principles.

Multiple principles, aggregated: Evaluate response pairs against multiple constitutional principles and aggregate the results. This gives more reliable signal than relying on any single principle. Different principles might favor different responses, and aggregation helps identify responses that perform well across multiple dimensions. This approach also gives resilience against any single principle being poorly specified or having unintended consequences. If one principle produces noisy judgments, the other principles can give the dominant signal.

Position debiasing: AI models can exhibit position bias, preferring whichever response appears first (or last). Running each comparison twice with swapped positions and averaging the results reduces this bias. Position bias is a well-documented phenomenon in language models, likely arising from patterns in training data where the first option in a list is often the default or recommended choice. By running comparisons in both orderings and requiring agreement, we filter out preferences that are driven by position rather than content quality.

Research has found substantial agreement between AI and human preferences, though the agreement varies by task type. Understanding when AI preferences are reliable and when they diverge from human judgment is needed for designing effective RLAIF systems.

For tasks with clear quality differences (one response is factually wrong, one follows instructions while the other doesn't), AI and human preferences agree strongly, often exceeding 80% agreement. These are cases where the quality signals are salient and unambiguous, making evaluation relatively straightforward for both humans and AI. For fine-grained judgments involving style preferences, humor, or subtle harmful content, agreement tends to be lower. These cases require implicit cultural knowledge, personal experience, or sensitivity to context that current models may lack.

Importantly, AI preferences are not necessarily worse than human preferences, just different. Human annotators disagree with each other at substantial rates, often 20-30% disagreement on borderline cases. AI preferences give a different but often complementary signal. The systematic nature of AI preferences can be advantageous: while individual humans vary in their standards and attention levels, AI evaluators apply the same criteria consistently across all comparisons. This consistency means that even if AI preferences are somewhat miscalibrated in absolute terms, they give a stable signal that reward models can learn from effectively.

Research from Google DeepMind on their RLAIF work showed that models trained with AI feedback achieved comparable or sometimes superior performance to those trained with human feedback, particularly on helpfulness metrics. This suggests that for many alignment objectives, AI feedback gives sufficient signal. The key insight is that perfect agreement with human preferences is not necessary for effective training. What matters is that the AI preferences capture enough of the relevant quality distinctions to guide the model toward better behavior. Imperfect but consistent signal can be very effective for training, as long as the errors are not systematically pointing in the wrong direction.

Not all comparisons have clear answers. Effective RLAIF systems need strategies for handling ambiguous cases where neither response is obviously better, or where the appropriate judgment depends on factors not captured in the comparison. Forcing judgments on ambiguous cases introduces noise into the training signal and may teach the reward model spurious correlations.

Confidence calibration: Ask the AI for a preference and a confidence level. Low-confidence judgments can be excluded from training or given lower weight. This approach recognizes that not all preference labels are equally reliable. By incorporating confidence into the training process, we can weight the loss function to emphasize high-confidence comparisons where the signal is cleaner.

Abstention: Allow the AI to indicate that two responses are roughly equal quality, rather than forcing a choice. Ties can be excluded from reward model training. This approach is particularly useful for cases where both responses are acceptable and the differences come down to stylistic preferences that should not be optimized strongly.

Ensemble evaluation: Use multiple AI evaluators (different prompts, different models) and only include comparisons where evaluators agree. Requiring consensus before accepting a preference label adds another reliability check. Disagreement among evaluators signals ambiguity that should not contribute to training.