Reward Modeling › Worked Example: Computing Preference Loss

Let's trace through the loss computation for a single preference pair to solidify understanding. Walking through a concrete numerical example makes the abstract formulas tangible and reveals how the loss function behaves in practice across different scoring scenarios.

Suppose we have a prompt asking for a simple explanation of gravity, with two candidate responses. Response A (the chosen response) gives a simple, accessible explanation using an everyday analogy. Response B (the rejected response) gives a technically correct but overly complex description using field equations. After training, our reward model produces the following scores:

• $r(x, y_w) = 2.1$ for the preferred response (A, the accessible explanation)
• $r(x, y_l) = 0.8$ for the rejected response (B, the technical description)

Step 1: Compute the reward difference.

where:

• $\Delta r$: the reward difference between the chosen and rejected responses
• $r(x, y_w)$: the reward score for the preferred response ($2.1$)
• $r(x, y_l)$: the reward score for the rejected response ($0.8$)

This positive difference of 1.3 indicates the model correctly believes the preferred response is better. The question is: how confident is this prediction, and how much loss does it incur?

Step 2: Convert to a preference probability via the sigmoid.

where:

• $P(y_w \succ y_l)$: the calculated probability that the model assigns to the correct preference
• $\sigma(1.3)$: the sigmoid function applied to the reward difference
• $0.786$: the resulting probability (approximately 78.6%)

The model assigns about 78.6% probability to the correct preference. This is a reasonably confident prediction. This reflects the moderately large reward margin of 1.3 points. The model is more likely than not to get this comparison right, but it is not highly certain.

Step 3: Compute the negative log-likelihood loss.

where:

• $\mathcal{L}$: the computed preference loss value
• $-\log(0.786)$: the negative log-likelihood of the correct preference

The loss of 0.241 is relatively small, showing the model is performing well on this example. For reference, a model that assigns exactly 50% probability (pure uncertainty) would incur a loss of $-\log(0.5) \approx 0.693$.

Step 4: Compare with an incorrect ranking scenario.

Now consider if the model had incorrectly scored the responses, assigning higher reward to the technical description:

• $r(x, y_w) = 0.5$ (accessible explanation, incorrectly ranked lower)
• $r(x, y_l) = 1.8$ (technical description, incorrectly ranked higher)

Following the same steps: the reward difference is $0.5 - 1.8 = -1.3$. The sigmoid gives $\sigma(-1.3) = 1/(1 + e^{1.3}) = 1/(1 + 3.669) \approx 0.214$. The loss is $-\log(0.214) \approx 1.54$.

Step 5: Compare the two loss values.

The loss of 1.54 for the incorrect ranking is more than six times larger than the 0.241 for the correct ranking. This asymmetry demonstrates how the loss heavily penalizes incorrect rankings while giving smaller gradients when the model is already correct. The gradient signal is strongest where the model needs improvement, naturally focusing the learning process on mistakes.

Step 6: Consider the gradient implications.

The gradient of the loss with respect to the reward difference is $\partial \mathcal{L} / \partial \Delta r = \sigma(-\Delta r) - 1 = -\sigma(\Delta r)(1 - \sigma(\Delta r))$. For the correct-ranking case ($\Delta r = 1.3$), this gradient has magnitude $0.786 \times 0.214 \approx 0.168$. For the incorrect-ranking case ($\Delta r = -1.3$), the gradient magnitude is $0.214 \times 0.786 \approx 0.168$. While the gradient magnitudes are the same (by symmetry of the sigmoid), the direction is reversed: for the correct ranking, the gradient pushes toward a larger margin; for the incorrect ranking, it pushes toward flipping the order entirely. The large loss difference between the two cases reflects the log transformation, which compresses the scale near 1 and expands it near 0, creating the six-fold difference in loss from equal-magnitude gradient contributions.

This demonstration illustrates that the preference loss is a ranking objective and a calibrated probabilistic one. The gradients it produces reflect both the direction of the error and the model's current confidence, creating a training signal that naturally focuses effort where the model most needs to improve.