Iterative Alignment › Online Preference Learning

The distinction between online and offline preference learning parallels the classic reinforcement learning distinction between on-policy and off-policy methods. This parallel is not merely superficial; the same basic trade-offs apply in both contexts, and understanding these trade-offs helps clarify when each approach is appropriate.

In offline reinforcement learning, an agent learns from a fixed dataset of experiences collected by a different policy, perhaps a human expert or an earlier version of the agent. The challenge is that the agent cannot explore: it must learn entirely from the experiences provided. In offline preference learning, the situation is analogous. The model learns from preference comparisons between responses that were generated by some other model, and it cannot generate new responses to get feedback on during training.

Online preference learning offers several benefits over offline approaches:

• Distribution matching: Training data always reflects the current policy's behavior
• Progressive difficulty: As the model improves, comparisons become more fine-grained
• Adaptive exploration: The model can explore regions of response space that earlier versions couldn't reach
• Reduced overfitting: Fresh data prevents memorizing fixed preference patterns

Distribution matching directly addresses the motivation for iterative alignment. When training data comes from the current policy, every preference comparison provides information about choices the model is making. There is no disconnect between what the model generates and what the training signal evaluates. This tight coupling ensures that the training signal remains relevant throughout the optimization process.

Progressive difficulty emerges naturally from online learning. In early iterations, the model might generate responses with obvious quality differences, making preference judgments easy. As the model improves, the differences between its responses become more subtle, requiring more careful evaluation. This natural curriculum exposes the model to increasingly sophisticated distinctions, pushing it toward finer-grained improvements that offline data might not capture.

The major drawback is cost. Online learning requires generating new responses and obtaining preferences continuously, which can be expensive when using human annotators. Each iteration requires training time, candidate response generation, and evaluation resources. For organizations with limited annotation budgets, this ongoing cost can be prohibitive.

When generating responses for online preference learning, several sampling strategies exist. They differ in sample efficiency and coverage as well as computational cost:

Best-of-N sampling: Generate $N$ responses per prompt, use a reward model to select the best and worst, then train on this pair. This is sample-efficient but requires a reward model. The approach maximizes the information content of each preference pair by creating a substantial quality gap between the compared responses. However, it relies on the reward model's accuracy, and any systematic biases in the reward model will be amplified through this selection process.

Diverse sampling: Use different temperatures or sampling strategies to generate varied responses. This keeps the preference pairs distributed across diverse regions of response space. Higher temperatures produce more varied outputs, potentially exposing the model to a broader range of quality levels. This approach reduces the risk of overfitting to a narrow slice of the output distribution but may produce many comparisons between clearly bad responses, which provide limited learning signal.

Targeted sampling: Focus on prompts where the model's current responses are uncertain or low-quality, similar to active learning approaches. By directing annotation effort toward the most informative examples, this strategy can achieve better sample efficiency than random sampling. The challenge lies in identifying which prompts will yield the most useful preference comparisons before collecting those comparisons.

The RLHF pipeline we covered earlier is inherently online: PPO samples from the current policy and updates based on reward model scores. We can convert this to online DPO by:

1. Generating pairs of responses from the current policy
2. Using the reward model to determine preferences (instead of giving scalar rewards)
3. Training with DPO instead of PPO

This hybrid approach, sometimes called reward model distillation or RLHaF (Reinforcement Learning from Human-AI Feedback), combines the stability of DPO with the online sampling of RLHF. The key insight is that a reward model can be used to generate preference labels rather than to provide direct reward signals. By converting scalar rewards into binary preference comparisons, we can use the simpler DPO training procedure while retaining the benefits of online data generation.

This conversion is practical because DPO is more stable and easier to tune than PPO. By using the reward model only to generate preference labels rather than to guide policy gradient updates, we sidestep many of the challenges that make RLHF difficult in practice.