DPO Concept › Limitations of DPO

Despite its advantages, DPO is not universally superior to RLHF. Several important scenarios favor the traditional approach, and understanding these limitations is needed for making good decisions in practice.

The most basic limitation of DPO is its offline nature. DPO trains on a fixed dataset of preference pairs collected before training begins. The distribution of responses in this dataset was determined by whatever model generated them, typically the supervised fine-tuned base model. As DPO training progresses, the policy shifts away from this base distribution. By the end of training, the policy may generate responses that look qualitatively different from anything in the training data. When this happens, the preference pairs no longer accurately capture what kinds of comparisons the model faces when it generates outputs. The model is being evaluated on a different distribution than the one it was trained on, and its behavior in the region of mismatch is unconstrained.

Think of this distribution shift problem as training a chess player by showing them games between beginners, then asking them to play against grandmasters. The preference data that told you "knight to E5 is better than pawn to A4 in this situation" does not help when you encounter positions that only arise in grandmaster-level play. RLHF handles this naturally because it generates training data from the current policy at every step. This keeps the training distribution keeps up with the policy's improving capabilities. DPO has no such mechanism without deliberate intervention.

The distribution shift problem also has a subtler manifestation: DPO can cause the absolute probability of both chosen and rejected responses to decrease together. If DPO trains on a dataset where all responses are mediocre, the model learns to prefer the slightly less mediocre response over the slightly more mediocre one. But in doing so, it might lower the probability of both responses, shifting probability mass toward responses that never appeared in the training data at all. Whether those out-of-distribution responses are good or bad is entirely unconstrained by the DPO objective. Monitoring the log-probability ratios separately for chosen and rejected responses during training is needed to catch this failure mode early.

A second significant limitation is the loss of explicit reward interpretability. An explicit reward model provides a standalone tool for understanding and debugging alignment. You can run the reward model on arbitrary inputs to understand what it values. You can probe it with adversarial examples to find edge cases. You can use it as a filter to score and rank model outputs at inference time, selecting the best response from multiple candidates through best-of-n sampling. You can monitor it during training to track whether reward hacking is occurring. DPO's implicit reward offers none of these affordances directly. You can compute the log-probability ratio $\beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ as a proxy for the implicit reward, but this is less intuitive and harder to interpret than a dedicated reward model's scalar output.

A third limitation is that DPO is poorly suited to iterative refinement workflows. Some of the most effective alignment pipelines involve training, generating new outputs, collecting human preferences on those outputs, and repeating. Each iteration, the preference data is collected from the most recently trained model. This keeps the training distribution stays current. This iterative approach naturally fits the RLHF framework where online generation is already part of the training loop. With DPO, iterating requires explicitly halting training, generating new outputs with the current model, collecting new preferences, and starting a new training run. This is possible, and the technique is called iterative DPO or online DPO, but it adds significant operational overhead. It also requires careful management of the reference model: if you update the reference model at each iteration, the implicit reward changes; if you keep the original reference, it may become increasingly irrelevant as the policy drifts.

A fourth limitation involves complex, multi-objective reward structures. Many real-world alignment scenarios involve multiple competing objectives: helpfulness, harmlessness, honesty, factual accuracy, conciseness. An explicit reward model makes it natural to specify the tradeoffs between these objectives, either by combining multiple reward heads or by training separate reward models and combining their scores. DPO learns whatever preference structure is implicit in the training data, which reflects the annotators' intuitions about how to balance these objectives. If you want to adjust the relative weights of helpfulness versus safety after training, RLHF lets you modify the reward model or its combination weights. With DPO, you would need to re-collect preference data with the new weighting and retrain from scratch.