PPO for Language Models › Key Parameters

The success of PPO training depends heavily on correctly setting its hyperparameters. Unlike supervised learning where the learning rate is often the primary concern, PPO for language models has multiple interacting parameters that require careful tuning. Understanding what each parameter controls is the first step toward debugging training failures.

The key parameters for PPO training are:

• clip_epsilon: The clipping threshold (typically 0.1 or 0.2) that constrains the policy update. Smaller values produce more conservative updates and can slow convergence. Larger values allow faster learning but risk instability. A common heuristic is to start with 0.2 and reduce it if training becomes unstable.
• kl_coef ($\beta$): Coefficient for the KL penalty term, controlling how closely the policy must stay to the reference model. This is perhaps the most important hyperparameter for preventing reward hacking. Higher values produce safer but slower learning. In practice, some implementations use an adaptive $\beta$ that increases when KL exceeds a target threshold and decreases when KL falls below it.
• value_coef: Weight for the value function loss in the total optimization objective. Balancing this against the policy loss affects how quickly the value function adapts relative to the policy. If the value function lags too far behind, advantage estimates become unreliable.
• gamma: Discount factor for future rewards. In RLHF, this is typically set to 1.0 since we care equally about all tokens in the response. Using a discount factor less than 1.0 would bias the model toward placing quality in early tokens rather than later ones, which is generally not desired.
• lam: The GAE smoothing parameter that balances bias and variance in advantage estimation. Values around 0.95 are standard. Lower values give more biased but lower-variance advantages; higher values give less biased but higher-variance advantages.