RLHF Pipeline › RLHF Pipeline

Reinforcement Learning from Human Feedback turns a base language model into an assistant that follows instructions and gives useful, truthful responses without causing harm. Building a capable language model through pretraining is only half the story. A model trained purely to predict the next token has learned to be a generalist text predictor, not a helpful collaborator. It will complete sentences, continue stories, and extend documents, but it has no concept of what "helpful" means, no understanding of what a user wants, and no instinct to avoid harmful outputs. Bridging that gap requires a fundamentally different kind of training signal, one that originates not from next-token prediction but from human judgment about what constitutes a good response.

The previous chapters introduced the individual components: the Bradley-Terry model for preference modeling, reward model architecture and training, and the PPO algorithm adapted for language models. Now we assemble these pieces into a complete training pipeline. The assembly matters as much as any individual component. Each stage creates preconditions that the next stage depends on, and misunderstanding those dependencies is one of the most common sources of failure in real RLHF implementations. A reward model trained on the wrong data distribution, or PPO optimization started from a poorly prepared initial policy, can derail the entire alignment effort despite each component working correctly in isolation.

Think of the RLHF pipeline as a three-act apprenticeship. In the first act, the apprentice studies examples of excellent work from master craftspeople, learning the expected format and quality standard. In the second act, an experienced judge observes many examples and builds an intuition for what separates excellent work from mediocre work, capturing that judgment in a reproducible scoring mechanism. In the third act, the apprentice iterates, generating work, receiving scores, and adjusting behavior to earn higher marks, all while staying grounded enough in the original training to remain coherent. The magic is that this apprenticeship produces a model whose behavior reflects human preferences even on prompts it has never seen before, generalizing a signal from thousands of comparisons into millions of novel situations.

The RLHF pipeline consists of three sequential stages: Supervised Fine-Tuning (SFT), Reward Model (RM) training, and PPO optimization. Each stage builds on the previous one, progressively shaping the model's behavior. This chapter walks through the complete pipeline, examining the design decisions at each stage, the hyperparameters that govern training stability, and the debugging techniques needed for successful alignment. We will also develop intuition for the failure modes that make RLHF notoriously difficult to get right in practice, building toward a mental model that helps you diagnose problems when they arise.

Understanding the complete pipeline also illuminates the motivation for alternatives like Direct Preference Optimization, which we will encounter in upcoming chapters. Many of the simplifications that DPO introduces are direct responses to known weaknesses in this three-stage RLHF pipeline, and appreciating those weaknesses helps you understand how DPO works and why it was designed the way it was.