PPO Algorithm › Limitations and Impact

PPO became a standard practical reinforcement learning method and later a foundation for aligning large language models with human preferences. It approximates trust region stability without the computational complexity of TRPO. Before PPO, reliable RL training required extensive expertise and environment-specific tuning. PPO lowered the implementation barrier for applying deep RL to new domains. Its explicit separation between rollout collection and policy optimization, simple clipping mechanism, and well-tested open-source implementations all contributed to its widespread adoption.

PPO has important limitations, however. The clipping mechanism provides only approximate trust region enforcement, so the policy can still drift significantly over many updates, especially with high learning rates or many optimization epochs. The clipping prevents the optimizer from exploiting large policy changes, but it does not prevent the policy from drifting gradually across many small steps. This drift becomes especially problematic in RLHF settings where maintaining proximity to the supervised fine-tuned base model is needed for response quality. A language model that drifts too far from its pre-trained weights can lose coherence, generate repetitive or degenerate text, or exploit weaknesses in the reward model without improving human-rated quality.

Sample efficiency is a persistent concern with PPO. The algorithm requires substantial environment or reward model interaction to learn effectively. Data becomes stale after a few update epochs, requiring fresh collection for continued improvement. For language models, each reward model query involves a full forward pass through a large neural network, making data collection expensive. This sample inefficiency is one of the primary motivations for direct alignment methods like Direct Preference Optimization (DPO), which we will explore in a later chapter. DPO sidesteps the RL optimization loop entirely by formulating alignment as a supervised learning problem over preference pairs, trading PPO's flexibility for much better data efficiency.

PPO inherits the challenges of the actor-critic framework. The value function must accurately estimate expected returns for meaningful advantage computation. In high-dimensional state spaces like natural language, where the state is a sequence of tokens that can be astronomically large, value estimation is extremely difficult. Noisy value estimates produce high-variance advantages, leading to unstable policy gradients even with the clipping mechanism in place. Researchers have addressed this in various ways: larger critic networks, separate training schedules for actor and critic, and more aggressive advantage normalization. None of these fully resolves the underlying difficulty of value estimation in language domains.

PPO also optimizes for the provided reward signal without accounting for reward model uncertainty. The reward model is itself a neural network trained on human preference data, and it will be wrong in many edge cases. PPO's objective is to maximize expected reward model score, not to maximize actual human preference. When the policy finds inputs that the reward model scores highly but humans would not, this is called reward hacking. The next chapter discusses how KL divergence penalties and reference model constraints mitigate this problem in RLHF: by penalizing the language model for deviating too far from a reference policy (typically the supervised fine-tuned base model), we constrain the optimization to regions where the reward model's predictions are likely to be reliable.

Finally, PPO's on-policy nature means that experience collected from an old policy becomes stale quickly. The multiple-epoch optimization within a single rollout helps, but eventually the probability ratios drift too far from 1 and clipping becomes binding everywhere. At that point, no more useful gradient signal remains in the batch, and a new rollout must be collected. This cycle of collect-update-discard is inherently less efficient than off-policy methods that maintain a replay buffer and can reuse experience indefinitely. Researchers have explored hybrid approaches, such as using importance sampling corrections to extend the usable lifetime of collected data, but these introduce their own complexity and instability.