RLHF Pipeline › Summary

The RLHF pipeline turns a pretrained language model into an aligned assistant through three sequential stages. Supervised Fine-Tuning creates a model that understands the instruction-following format and produces reasonable responses. Reward Model training captures human preferences in a learnable function that provides optimization signal. PPO Fine-Tuning then optimizes the policy to maximize rewards while staying close to the reference model.

Key takeaways from this chapter:

• SFT provides the foundation: PPO requires a model that already produces usable responses; skipping SFT leads to unstable training
• Reward model quality constrains the policy: A flawed reward model will lead to flawed policies; validate carefully before PPO
• KL penalty prevents catastrophic drift: Without anchoring to the reference model, the policy will exploit reward model weaknesses
• Monitor multiple metrics: Single metrics can be misleading; track reward, KL, entropy, and response characteristics together
• Debugging requires examining actual outputs: Metrics summarize behavior, but reading generated responses reveals problems that numbers hide
• The pipeline has compounding failure modes: Errors in SFT propagate to reward model training, and errors in the reward model propagate to the PPO policy

The RLHF pipeline established the template for aligning large language models, but its complexity and cost have motivated simpler alternatives. The next chapter examines the KL divergence penalty in mathematical detail, followed by chapters on Direct Preference Optimization, which eliminates the reward model and PPO stages entirely while achieving comparable alignment results.