PPO Algorithm › Key Parameters
Michael BrenndoerferDecember 27, 20251 min read
Language AI HandbookAll chapters Part 37 of 67Alignment and RLHF
PPO Algorithm › Key ParametersOne section · Sections in this chapter11 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for PPO are:
- clip_epsilon: The clipping parameter that defines the trust region width (typically 0.1 to 0.3). Smaller values provide tighter constraints on policy updates, while larger values allow more aggressive changes.
- gae_lambda: Controls bias-variance tradeoff in advantage estimation (typically 0.95 to 0.99). Higher values use longer horizons for advantage computation.
- n_epochs: Number of optimization epochs per rollout (typically 3 to 10). More epochs extract more learning from each batch but risk overfitting to old data.
- batch_size: Minibatch size for gradient updates (typically 32 to 256). Larger batches provide more stable gradients.
- rollout_length: Number of timesteps to collect before updating (typically 2048 for simple tasks). Longer rollouts provide better advantage estimates.
- value_coef: Coefficient for value function loss in total objective (typically 0.5). Controls how much weight to give value function training relative to policy training.
- entropy_coef: Coefficient for entropy bonus (typically 0.01). Higher values encourage more exploration.