DPO Concept › The Role of Beta

The parameter $\beta$ plays a useful role in both RLHF and DPO, but it is worth understanding its specific effect in the DPO context in detail. Recall that $\beta$ controls the strength of the KL constraint, mediating the tradeoff between matching preferences and staying close to the reference model. This single parameter has large effects on both the optimization dynamics and the final solution, and choosing it well requires understanding its role.

Think of $\beta$ as the "stubbornness" of the model toward its original training. A small $\beta$ means the model is willing to change substantially based on the preference data: it will move far from the reference distribution to match human preferences. A large $\beta$ means the model is conservative: it insists on staying close to how it was originally trained, making only modest adjustments to satisfy the preference data. The right value depends on how much you trust the preference data relative to the pretraining distribution.

In RLHF, we explicitly compute the KL divergence between the policy and reference model and add it as a penalty term with coefficient $\beta$. This requires computing log-probabilities under both models and combining them appropriately. In DPO, the constraint is built into the loss function itself through the log-ratio terms. The $\beta$ parameter determines how aggressively the model should diverge from the reference when the preference signal demands it. Both approaches implement the same mathematical constraint, but DPO does so implicitly through the structure of the loss.

Low $\beta$ values allow the model to deviate significantly from the reference to match preferences. When $\beta$ is small, the preference margin has a large effect on the loss, so the model will make substantial changes to ensure it assigns higher probability to preferred responses. This can lead to better preference satisfaction but risks overfitting to the preference dataset or moving too far from the model's pretrained capabilities. A model trained with very low $\beta$ might become extremely good at the specific types of comparisons in the training data while losing the general capabilities it had before alignment. In the limit of $\beta \to 0$, the model collapses entirely to a distribution concentrated on whatever responses appear as "chosen" in the training data.

High $\beta$ values keep the model close to the reference. This provides regularization but can limit the model's ability to fully incorporate preference information. When $\beta$ is large, even large preference margins produce relatively modest gradients, so the model changes slowly and stays close to its starting point. The optimal choice depends on the quality and coverage of the preference dataset, the size of the model, and the desired balance between helpfulness and safety. Practitioners often need to tune $\beta$ empirically, though values between 0.1 and 0.5 are common starting points for most alignment tasks. Very small values like 0.01 risk degradation of general capabilities, while very large values like 2.0 may fail to align the model.