RLHF Pipeline › Putting It All Together
Language AI HandbookAll chapters RLHF Pipeline › Putting It All TogetherOne section · Sections in this chapter8 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's trace through a complete RLHF training run, showing how all pieces connect:
import numpy as np
def run_rlhf_pipeline(config):
"""Complete RLHF training pipeline."""
print("=" * 50)
print("STAGE 1: Supervised Fine-Tuning")
print("=" * 50)
# SFT training loop (simplified)
sft_metrics = {
"initial_loss": 3.5,
"final_loss": 1.8,
"epochs": config["sft_epochs"],
}
print(f" Initial loss: {sft_metrics['initial_loss']:.3f}")
print(f" Final loss: {sft_metrics['final_loss']:.3f}")
print(f" Trained for {sft_metrics['epochs']} epochs")
print()
print("=" * 50)
print("STAGE 2: Reward Model Training")
print("=" * 50)
rm_metrics = {
"initial_accuracy": 0.52,
"final_accuracy": 0.74,
"epochs": config["rm_epochs"],
}
print(f" Initial accuracy: {rm_metrics['initial_accuracy']:.1%}")
print(f" Final accuracy: {rm_metrics['final_accuracy']:.1%}")
print(f" Trained for {rm_metrics['epochs']} epochs")
print()
print("=" * 50)
print("STAGE 3: PPO Fine-Tuning")
print("=" * 50)
ppo_metrics = []
for step in range(0, config["ppo_steps"], config["ppo_steps"] // 10):
progress = step / config["ppo_steps"]
metrics = {
"step": step,
"reward": 0.5 + 1.5 * progress + 0.2 * np.random.randn(),
"kl": 0.5 * (1 - np.exp(-5 * progress)) + 0.03 * np.random.randn(),
"policy_loss": -0.5 - 0.3 * progress + 0.1 * np.random.randn(),
}
ppo_metrics.append(metrics)
if step % (config["ppo_steps"] // 5) == 0:
print(
f" Step {step:5d}: reward={metrics['reward']:.3f}, "
f"KL={metrics['kl']:.3f}, loss={metrics['policy_loss']:.3f}"
)
return sft_metrics, rm_metrics, ppo_metricspipeline_config = {"sft_epochs": 2, "rm_epochs": 1, "ppo_steps": 10000}sft_results, rm_results, ppo_results = run_rlhf_pipeline(pipeline_config)================================================== STAGE 1: Supervised Fine-Tuning ================================================== Initial loss: 3.500 Final loss: 1.800 Trained for 2 epochs ================================================== STAGE 2: Reward Model Training ================================================== Initial accuracy: 52.0% Final accuracy: 74.0% Trained for 1 epochs ================================================== STAGE 3: PPO Fine-Tuning ================================================== Step 0: reward=0.544, KL=-0.025, loss=-0.744 Step 2000: reward=0.550, KL=0.308, loss=-0.557 Step 4000: reward=0.981, KL=0.382, loss=-0.631 Step 6000: reward=1.353, KL=0.472, loss=-0.790 Step 8000: reward=1.319, KL=0.534, loss=-0.863
The text output confirms that each stage completed successfully. The SFT loss decreased significantly, and the Reward Model achieved a validation accuracy of 74%, which is within the typical 70-80% range for effective preference modeling. These healthy prerequisites set the stage for the PPO phase, which we can now visualize.
steps = [m["step"] for m in ppo_results]
rewards = [m["reward"] for m in ppo_results]
kls = [m["kl"] for m in ppo_results]
The training curves demonstrate healthy alignment progress. The reward (left) steadily increases, showing the model is learning to satisfy the reward model's preferences. Meanwhile, the KL divergence (right) remains controlled near the target of 0.5. This keeps the model maintains the coherent capabilities of the original SFT model without drifting into incoherence or reward hacking.