RLHF Pipeline › The Three-Stage Pipeline
Language AI HandbookAll chapters RLHF Pipeline › The Three-Stage PipelineOne section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The RLHF pipeline follows a specific sequence that turns a pretrained language model into an aligned assistant. Understanding why each stage exists and how they connect is important for successful implementation. Each stage has a distinct purpose, and the stages form a dependency chain where the quality of each stage's output places a ceiling on everything that follows.
The pipeline is a technical recipe built on assumptions about how alignment works and what data is available. The stages are ordered the way they are because of a specific logic: you cannot collect useful preference data without responses to compare, you cannot train a reward model without preference data, and you cannot run stable PPO without a reward model. The forward arrow of causality runs through the entire pipeline, and this is why short-circuiting any stage tends to undermine subsequent ones even when the skipped stage seems redundant.
A second important dimension is that all three models in the pipeline, the SFT model, the reward model, and the PPO policy, typically share the same underlying architecture. They are all transformer language models of roughly the same parameter count, often initialized from the same pretrained weights. This shared foundation means that improvements in base model capability propagate through all three stages simultaneously. When OpenAI upgraded from GPT-3 to GPT-4 as the base, the improvements benefited every stage of the alignment pipeline, including the final PPO policy. The alignment procedure is more like a specialization of capabilities that already exist than an injection of entirely new capabilities.
Stage 1: Supervised Fine-Tuning (SFT) takes a pretrained language model and trains it on high-quality demonstrations of desired behavior. This stage teaches the model the format and style of helpful responses, creating a starting point that can already follow instructions reasonably well.
Stage 2: Reward Model Training creates a model that predicts human preferences. Using comparison data where humans ranked alternative responses, the reward model learns to assign scalar scores which reflects response quality. As we covered in the Reward Modeling chapter, this model provides the optimization signal for the final stage.
Stage 3: PPO Fine-Tuning optimizes the SFT model to maximize rewards from the reward model while staying close to its original behavior. Building on the PPO for Language Models chapter, this stage performs the actual alignment through reinforcement learning.
Why the Order Cannot Be Reversed
The sequential nature of the pipeline is not arbitrary, and understanding the causal logic makes it far easier to reason about what can go wrong. SFT must precede reward model training because the reward model needs realistic candidate responses to compare. If you attempted to collect preference data from a raw pretrained model, the responses would be so incoherent and off-format that human annotators would struggle to determine which was "better" in any meaningful sense. Both responses might simply be unusable, making the comparison uninformative.
Reward model training must precede PPO because PPO requires a dense, differentiable reward signal. The alternative, having humans rate every generated response during PPO training, would be impractically slow. PPO generates millions of samples during training; each requiring a human rating would make the procedure orders of magnitude too expensive. The reward model acts as a learned proxy that can be evaluated in milliseconds rather than hours.
PPO must come last because it requires a stable starting point and a clear optimization target. Without the SFT model as initialization, the policy gradient updates would struggle to produce coherent language. Without the reward model as the optimization target, there is no well-defined objective to optimize toward. The PPO stage consumes all the infrastructure built in the previous two stages and converts it into the final aligned model.