Reward Modeling › Preference Loss Function

The reward model is trained on human preference data, where annotators indicate which of two responses they prefer for a given prompt. We need a loss function that encourages the model to assign higher rewards to preferred responses. The key insight is that we do not need absolute quality labels. Instead, we only need relative comparisons, and from these pairwise signals, the model can learn a consistent scoring function.

Think of this like training a judge to rank competitors. You do not need to know that competitor A deserves a score of 7.4 out of 10. You only need to know that A is better than B, and from enough of these comparisons, the judge gradually develops an internal scale. The Bradley-Terry preference loss formalizes this intuition: each comparison example provides a gradient signal that nudges the scoring function to separate the preferred response from the rejected one, and the accumulated effect of many such nudges produces a globally consistent ranking.

The preference loss operates entirely in the space of reward differences, not absolute values. This shift-invariance has a practical implication: you never need to define what a "good" absolute reward score looks like, and you never need to normalize rewards during training. The model learns to separate preferred from rejected responses regardless of where on the number line those scores happen to fall. Two reward models that produce identical rankings but different absolute values are functionally equivalent for the purpose of RL training, and the loss function makes no distinction between them.

This shift-invariance also means that the absolute magnitude of reward scores is not meaningful on its own. A reward of 3.7 says nothing about quality in isolation. What matters is whether 3.7 is higher or lower than the reward assigned to alternative responses for the same prompt. The reward model's job is to learn an ordering, not a cardinal scale. However, the downstream RL algorithm often benefits from reward signals that are normalized to a consistent range, which is why reward normalization is applied as a post-processing step after training.

The mathematical foundation for our loss function comes from the Bradley-Terry model, which provides a principled way to convert scalar scores into preference probabilities. As we established in the Bradley-Terry chapter, the probability that response $y_w$ (the winner) is preferred over response $y_l$ (the loser) given their reward scores is:

$$P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))$$

where:

• $P(y_w \succ y_l | x)$: the probability that response $y_w$ is preferred over $y_l$ given prompt $x$
• $x$: the prompt text input
• $\sigma$: the logistic sigmoid function, $\sigma(z) = \frac{1}{1+e^{-z}}$
• $r(x, y)$: the scalar score output by the reward model for a given input pair
• $y_w, y_l$: the winning and losing responses, respectively

This equation has a beautiful interpretation. The preference probability depends only on the difference between rewards, not their absolute values. If one response scores 10 points higher than another, the preference probability is the same whether the scores are (15, 5) or (105, 95). This shift-invariance is both mathematically convenient and practically useful: it means the model only needs to learn relative quality, not calibrate to some arbitrary absolute scale.

The sigmoid function $\sigma$ turns this difference into a valid probability between 0 and 1. When the reward difference is zero, both responses are equally preferred with probability 0.5. As the difference grows positive (the winner scores much higher), the probability approaches 1. As it grows negative (the winner scores lower, showing a model error), the probability approaches 0.

To train the model, we maximize the log-likelihood of the observed preferences. This corresponds to minimizing the negative log-likelihood, which is equivalent to the binary cross-entropy loss applied to the reward difference. The derivation proceeds through several algebraic steps, each illuminating a different aspect of the loss function. We can derive the final loss form step-by-step:

$$\begin{aligned} \mathcal{L} &= -\log P(y_w \succ y_l | x) && \text{(negative log-likelihood)} \\ &= -\log \sigma(r(x, y_w) - r(x, y_l)) && \text{(substitute probability model)} \\ &= -\log \left( \frac{1}{1 + e^{-(r(x, y_w) - r(x, y_l))}} \right) && \text{(substitute sigmoid definition)} \\ &= \log(1 + e^{-(r(x, y_w) - r(x, y_l))}) && \text{(log property: } -\log(1/z) = \log z \text{)} \\ &= \log(1 + e^{r(x, y_l) - r(x, y_w)}) && \text{(simplify exponent)} \end{aligned}$$

where:

• $\mathcal{L}$: the preference loss to be minimized
• $P(y_w \succ y_l | x)$: the probability that response $y_w$ is preferred
• $\sigma$: the logistic sigmoid function
• $x$: the prompt text input
• $y_w, y_l$: the winning and losing responses
• $r(x, y_l) - r(x, y_w)$: the reward difference (negative when the preferred response scores higher, as desired)
• $e^{(\cdot)}$: the exponential function
• $\log$: the natural logarithm

The final form of the loss, $\log(1 + e^{r(x, y_l) - r(x, y_w)})$, is known as the softplus function applied to the negative reward margin. This form is numerically stable and commonly implemented directly in deep learning frameworks as F.logsigmoid(reward_diff).neg() or equivalently via the binary cross-entropy function with logits.

The main point behind this loss is that it turns a ranking problem into a maximum likelihood problem. Instead of asking "is A ranked above B?", it asks "what is the probability that A is preferred over B, and how do we maximize that probability?" This probabilistic framing enables smooth gradient-based optimization and naturally handles noisy or inconsistent annotations by letting the model to express uncertainty through reward differences that are neither very large nor very small.

This equation has a clear interpretation that connects directly to how we want the model to behave. Consider what happens in different scenarios.

When the reward model correctly assigns a much higher score to the preferred response ($r(x, y_w) \gg r(x, y_l)$), the difference is a large positive number, and $\sigma$ of that difference approaches 1. Taking the negative log gives a loss close to zero. The model has confidently made the right prediction, so there is little left to learn from this example.

Conversely, when the model incorrectly assigns a higher score to the rejected response, the difference is negative, $\sigma$ outputs a value near 0, and the negative log produces a large loss. This large loss creates strong gradients that push the model away from its incorrect belief.

The gradient of this loss pushes the model to:

• Increase $r(x, y_w)$ (the preferred response's reward)
• Decrease $r(x, y_l)$ (the rejected response's reward)

The magnitude of these updates depends on how confident the model currently is. If the model already strongly prefers the correct response, gradients are small. If it's uncertain or wrong, gradients are larger. This is the standard behavior of log-loss functions and provides natural calibration during training. The model learns most from examples where it is wrong or uncertain, while examples it already handles well contribute minimally to parameter updates.

One subtle property of the preference loss is that it has no natural floor for the margin. Even if the model correctly ranks a pair with a margin of 100, the loss is not exactly zero; it approaches zero but never reaches it. This means training never "stops" in the sense of achieving exactly zero loss on the training data, which acts as a mild form of implicit regularization. The model is always being pushed to increase the margin, which encourages confident predictions, but the diminishing gradient ensures this pressure decreases as confidence grows.

In practice, we train on batches of preference pairs rather than individual examples. For a dataset of $N$ preference pairs $\mathcal{D} = \{(x^{(i)}, y_w^{(i)}, y_l^{(i)})\}_{i=1}^N$, the full training objective is:

$$\mathcal{L}(\theta) = -\frac{1}{N} \sum_{i=1}^N \log \sigma\left(r_\theta(x^{(i)}, y_w^{(i)}) - r_\theta(x^{(i)}, y_l^{(i)})\right)$$

where:

• $\mathcal{L}(\theta)$: the total loss over the dataset parameterized by model weights $\theta$
• $N$: the total number of preference pairs in the batch
• $\sum_{i=1}^N$: summation over all examples in the batch
• $x^{(i)}$: the prompt for the $i$-th example
• $y_w^{(i)}, y_l^{(i)}$: the preferred and rejected responses for the $i$-th example
• $r_\theta$: the reward model function parameterized by $\theta$
• $\sigma$: the sigmoid function converting the score difference into a probability
• $\log$: the natural logarithm

The averaging by $N$ normalizes the loss across different batch sizes. This keeps learning rates behave consistently regardless of batch configuration. This is the standard reward modeling loss used in systems like InstructGPT, Anthropic's Constitutional AI, and most open-source RLHF implementations. Its simplicity and effectiveness have made it the default choice for training reward models from pairwise preferences.

From a statistical perspective, minimizing this loss is equivalent to maximum likelihood estimation of the Bradley-Terry model parameters. We are finding the reward function $r_\theta$ that maximizes the joint probability of observing all the preference comparisons in our dataset, under the assumption that preferences follow the Bradley-Terry model. This principled probabilistic foundation distinguishes the preference loss from ad hoc margin-based objectives and ensures the resulting reward model has well-defined calibration properties.

Some implementations add a margin term to encourage larger reward differences between preferred and rejected responses:

$$\mathcal{L} = -\log \sigma(r(x, y_w) - r(x, y_l) - m)$$

where:

• $\mathcal{L}$: the margin-based preference loss
• $x$: the prompt text
• $y_w, y_l$: the preferred and rejected responses
• $m > 0$: a fixed margin hyperparameter so the winner's score exceeds the loser's by at least $m$
• $r(x, y_w) - r(x, y_l)$: the raw reward difference
• $\sigma$: the sigmoid function
• $\log$: the natural logarithm

The margin $m$ acts as a buffer zone. With a margin of, say, 0.5, the model is penalized unless the preferred response scores at least 0.5 points higher than the rejected one. This prevents the model from being satisfied with arbitrarily small reward differences, even when the ranking is technically correct.

This penalizes the model even when it correctly ranks preferences but with a small margin, encouraging more confident predictions. The idea is that a reliable reward model should produce clearly separated scores. This makes it easier for downstream RL algorithms to distinguish good from bad responses. However, this can hurt calibration and is not universally used. The margin effectively changes the decision boundary from zero to $m$, which may not align with the true underlying preference probabilities.

An adaptive variant uses a learned or example-specific margin based on annotator confidence. If multiple annotators rated a comparison and they agreed strongly, the margin is larger. If they disagreed or the comparison was rated as close, the margin is smaller. This approach more accurately reflects the difficulty of each comparison and can lead to better-calibrated reward models, but requires richer annotation metadata that is not always available.