DPO Derivation › Key Parameters

The key parameter for DPO is:

• beta: The temperature parameter (often denoted as $\beta$) that scales the log-ratio of the policy and reference probabilities. It controls the strength of the KL divergence penalty, with larger values keeping the policy closer to the reference model.

Choosing $\beta$ in practice requires balancing two failure modes. If $\beta$ is too small, the DPO loss will push the policy aggressively in the direction of human preferences without adequate constraint from the reference model. This can cause the policy to drift far from the reference distribution and start generating fluent-but-wrong outputs, such as responses that contain the right topics but with incorrect factual content, or responses that are overly verbose because longer responses were slightly preferred in the training data. If $\beta$ is too large, the policy can barely move away from the reference, which means it learns slowly and may not improve much even after thousands of training steps. Empirically, values in the range 0.01 to 0.5 are common, with 0.1 being a popular default starting point.