PPO for Language Models › Limitations and Challenges

Applying PPO to language models, while effective, comes with significant challenges that require careful navigation. Understanding these limitations helps explain why the field has explored alternative methods and also sets realistic expectations for what PPO can and cannot achieve.

Sparse rewards make credit assignment fundamentally difficult. When a complete response receives a reward, determining which tokens contributed positively and which detracted is imprecise at best. The advantage function provides some differentiation through the value function's predictions, but it operates through imperfect estimates. The value function can only learn what patterns in intermediate states correlate with final rewards; it cannot directly observe the causal relationships between specific word choices and response quality. A model might learn that responses beginning with "Certainly! Here is..." receive high rewards, without learning anything about the actual quality of what follows. This imprecision can lead to slow learning, gaming of surface patterns, and inconsistent behavior across similar prompts.

The high computational cost of PPO creates a significant barrier to adoption and iteration. PPO requires generating complete responses during training, which is much slower than the teacher-forcing paradigm used in supervised learning. During teacher forcing, the model processes all tokens in parallel because it always sees the correct previous token. During generation, tokens must be produced sequentially. Each training step involves running the policy model to generate text, running it again to compute log probabilities, running the reference model for KL computation, running the value model for advantage estimation, and running the reward model for scoring. This pipeline is sequential in several places and requires keeping multiple large models in memory simultaneously. A single PPO training run can cost 10 to 100 times more compute than the equivalent supervised fine-tuning run, making extensive hyperparameter search impractical.

The sensitivity to hyperparameters creates both reproducibility challenges and practical fragility. The clipping parameter, KL coefficient, learning rate, batch size, number of PPO epochs, advantage normalization, and many other choices all interact in complex ways. Settings that work well for one model size, one task, or one reward model may fail on another configuration. Small changes to any hyperparameter can lead to training instability, where the policy diverges or collapses, or to reward hacking, where the policy finds degenerate solutions. This sensitivity is one of the primary motivations for exploring simpler alternatives like Direct Preference Optimization (DPO), which we will discuss in an upcoming chapter, that eliminate many of these hyperparameters by eliminating the RL training loop entirely.

The reliance on a learned reward model introduces all the challenges we discussed in the Reward Hacking chapter, and PPO's optimization pressure exacerbates them. The reward model is a proxy for human preferences, and optimizing it too aggressively can lead to responses that score highly according to the model while being less useful or even harmful. The reward model was trained on a finite dataset of human comparisons and generalizes imperfectly to the full distribution of prompts the policy will encounter. In the regions of prompt space that were underrepresented in the reward model training data, the reward model may give unreliable signals that the policy will happily exploit. The KL penalty mitigates but does not eliminate this risk, because a sufficiently flexible policy can find high-reward behaviors within the KL constraint that still fail human evaluation.

Finally, the training distribution mismatch between the supervised fine-tuning phase and the RL phase can create instabilities. The SFT model was trained to predict specific human-written responses. The RL policy must generate diverse responses that receive high reward, which may include phrasings and styles very different from the SFT training data. This distribution shift means that the value function must extrapolate far from its training distribution when evaluating novel responses, potentially giving noisy advantage estimates that destabilize the policy update. Some practitioners address this by warming up the value function with supervised training before starting RL, or by periodically refreshing the reference model to track the evolving policy.