RLAIF › Worked Example: Tracing a Single Comparison Through the CAI Pipeline

To make the abstract mechanics concrete, let's trace a single comparison through the CAI preference generation pipeline step by step. This worked example uses a numerical trace so you can verify each step yourself.

Setup: We have a prompt "What is gradient descent?" and two candidate responses.

• Response A (100 characters): "Gradient descent is an optimization algorithm that iteratively adjusts parameters to minimize a loss function by following the negative gradient."
• Response B (12 characters): "Math stuff."

We apply three constitutional principles with weights 1.5, 1.2, and 1.0.

Step 1: Score each response using length and content signals.

For Response A: base score = 100 * 0.01 = 1.00. Response A contains reasoning keywords ("by following"), so add 0.2. No uncertainty markers. Score A = 1.20.

For Response B: base score = 12 * 0.01 = 0.12. Response B is under 50 characters, so subtract 0.5. Score B = -0.38.

Step 2: Compute the score difference and confidence for each principle.

The absolute difference is |1.20 - (-0.38)| = 1.58. Confidence = min(0.95, 0.5 + 1.58) = 0.95. Since Score A > Score B, the choice is "A" for all three principles.

Step 3: Apply position debiasing.

We run the comparison again with responses swapped (B first, A second). The same scoring logic applies: now "B" in the flipped run is the original Response A, which still scores much higher. The flipped run returns choice "B", which we flip back to "A". Both orderings agree, so the label passes the position bias filter.

Step 4: Aggregate across principles using weights.

Votes for A: (1.5 * 0.95) + (1.2 * 0.95) + (1.0 * 0.95) = 1.425 + 1.14 + 0.95 = 3.515. Total weight = 3.7. Vote share for A = 3.515 / 3.7 = 0.950. Average confidence = 0.95.

Step 5: Apply the confidence threshold.

Confidence (0.95) exceeds the threshold (0.6). Vote share (0.950) exceeds 0.5. The label is accepted with Response A as "chosen" and Response B as "rejected."

Step 6: What the reward model learns.

This label tells the reward model that, for a prompt about gradient descent, a detailed technical explanation should receive a higher reward score than a dismissive non-answer. After training on many such labels, the reward model learns to assign high rewards to responses that address the question in detail and low rewards to evasive or empty responses.

The key insight is that the confidence score (0.95) is high because the quality difference is large and unambiguous. Had both responses been reasonable but different in style, the confidence would have been lower, and the label might have been abstained. This filtering ensures that only clear, reliable quality distinctions make it into the training signal.