Policy Gradient Methods › Limitations and Practical Challenges

REINFORCE, while theoretically elegant and mathematically well-grounded, has significant practical limitations that make it challenging to apply directly to large language model alignment. Understanding these limitations is important both for appreciating why more sophisticated algorithms exist and for knowing when simpler approaches might be appropriate.

Sample efficiency. REINFORCE requires many samples to produce reliable gradient estimates. For large language models generating hundreds of tokens, each sample is expensive: it requires a full autoregressive forward pass through a model with billions of parameters, potentially followed by a separate reward model evaluation. Training to convergence can require millions of sampled sequences. At a cost of perhaps several seconds per sample on expensive hardware, the total compute required becomes prohibitive. The $O(1/N)$ variance reduction from batching means you need 100 times more samples just to reduce variance by a factor of 10 compared to having a perfect gradient estimate.

Credit assignment at the token level. The credit assignment problem is only partially solved. REINFORCE attributes equal credit to all tokens in a sequence based on the sequence-level reward. This ignores the reality that different tokens contribute differently to response quality. A response might be mostly excellent with one factually incorrect sentence, yet every token in the response receives the same blame for that error. The model cannot learn to avoid that specific error while keeping the rest of its behavior intact. More sophisticated algorithms like actor-critic methods attempt finer-grained credit assignment by training a value function that estimates the expected future reward from each intermediate state.

Update stability. REINFORCE provides no guarantees about the size of policy updates. A lucky high-reward sample can push the policy dramatically in one direction, only for the next batch to push it back in the opposite direction. This oscillation wastes computation and can destabilize training. More seriously, if the policy changes too much in one update, future samples will be drawn from a different distribution, making old gradient estimates less relevant. This mismatch between the policy at sample time and the policy at update time can compound over multiple steps, leading to divergence. The PPO algorithm, covered in the next chapter, directly addresses this by constraining how much the policy can change per update.

On-policy requirement and sample reuse. Policy gradient methods are on-policy: the gradient estimate is valid only for the policy that generated the samples. Once we update the policy, the samples become stale, and we must generate new ones. This is in contrast to off-policy methods (like Q-learning) that can reuse old samples from a replay buffer. The on-policy requirement means we cannot reuse samples across multiple gradient steps, further reducing sample efficiency. Every parameter update requires generating a fresh batch of sequences.

Despite these limitations, REINFORCE remains important as the foundational algorithm for understanding all more sophisticated policy gradient methods. PPO, TRPO, and other production algorithms all build on the same core gradient estimator derived in this chapter. Understanding REINFORCE's mechanics, failure modes, and the mathematical properties that enable variance reduction is prerequisite knowledge for understanding why those algorithms make the design choices they do. The log-derivative trick, the zero-mean score function, the baseline property, and the causal structure of reward-to-go all appear in more advanced algorithms, just in more sophisticated forms.