PPO Algorithm › Key Hyperparameters

PPO's performance depends on several key hyperparameters, and understanding what each one controls is needed for successful application. These hyperparameters interact with each other, so tuning one often requires adjusting others. The values below represent good starting points, but every domain benefits from some tuning.

The clipping parameter $\epsilon$ is the most basic hyperparameter because it directly controls the trust region. Smaller values make PPO more conservative, potentially requiring more rollouts to achieve the same policy improvement. Larger values allow faster learning but increase the risk of instability. The number of update epochs per rollout determines how much you squeeze out of each collected batch. More epochs improve sample efficiency (less environment interaction needed) but increase the risk that the policy drifts too far from the old policy within a single rollout. PPO's clipping limits this drift, but with many epochs and aggressive learning rates, the policy can still move substantially.

The key hyperparameters and their typical values are:

• Clip parameter ($\epsilon$): Controls policy change by defining the trust region as $[1-\epsilon, 1+\epsilon]$ for the probability ratio. Values of 0.1 to 0.3 are typical. Smaller values (e.g., 0.05) constrain updates too tightly and slow learning. Larger values (e.g., 0.5) provide insufficient constraint and risk instability.
• GAE lambda ($\lambda$): Bias-variance tradeoff for advantage estimation. Values of 0.95 to 0.99 are typical.
• Number of epochs: Number of passes through collected data. Using 3 to 10 epochs balances sample efficiency against overfitting to old data.
• Minibatch size: Larger batches provide more stable gradients but may overfit. Sizes of 32 to 256 are common.
• Rollout length: How much data to collect before updating. Longer rollouts provide better advantage estimates but slower iteration.
• Learning rate: Values of 1e-4 to 3e-4 are typical for PPO. You can use learning rate scheduling to improve convergence.

Language model implementations often require adjusted values. The next chapter explores LLM-specific considerations, including how to adapt the reward signal from a reward model, manage the KL penalty against the reference model, and handle the much larger action spaces of language generation.