PPO Algorithm › The Problem with Vanilla Policy Gradients

The instability in vanilla policy gradient methods is not a superficial implementation problem that can be fixed with a better learning rate schedule. It is a structural problem rooted in how the gradient signal relates to policy performance. To understand why PPO's solution is the right one, we need to be precise about exactly what goes wrong.

Recall that the policy gradient takes the form:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot A_t\right]$$

To understand why this formula creates practical difficulties, examine what each component contributes:

• $\nabla_\theta J(\theta)$: the gradient of the expected cumulative reward with respect to policy parameters, showing the direction to adjust parameters to increase expected reward
• $\mathbb{E}_{\tau \sim \pi_\theta}$: expectation over trajectories sampled from the current policy $\pi_\theta$
• $\theta$: parameters of the policy network that we are optimizing
• $J(\theta)$: the expected cumulative reward under policy $\pi_\theta$
• $\tau$: a trajectory (sequence of states and actions) sampled from the current policy
• $\pi_\theta(a_t | s_t)$: the probability of taking action $a_t$ in state $s_t$ under the current policy
• $T$: the time horizon, the length of the episode
• $A_t$: the advantage function at time $t$, estimating how much better action $a_t$ is compared to the average action in state $s_t$
• $\nabla_\theta \log \pi_\theta(a_t | s_t)$: the gradient of the log probability with respect to policy parameters, showing the direction to adjust parameters to make action $a_t$ more likely
• $\sum_{t=0}^{T}$: sum over all timesteps in the trajectory from 0 to T

The basic issue with this formulation is that the gradient provides no guidance about step size. The policy gradient theorem tells us which direction to move parameters to increase expected reward, but it remains entirely silent about how far we should move in that direction. This is analogous to knowing that walking north takes you closer to your destination but not knowing whether to take one step or one hundred steps. A step in the gradient direction improves performance locally, within an infinitesimally small neighborhood around the current parameters. However, nothing in the mathematics prevents taking such a large step that you overshoot into a region where the policy performs terribly. Policy performance is not smooth across the parameter space: a policy that seems promising can suddenly fail after a large update.

This problem is especially acute because of three interconnected challenges:

• Non-stationary data distribution: The policy generates its own training data. When the policy changes significantly, the state distribution also changes, potentially invalidating previously learned value estimates.
• High variance: Policy gradient estimates are inherently noisy, which makes it difficult to distinguish signal from noise.
• Irreversibility: A bad update might move the policy to a region where it never encounters states that would help it recover.

Consider a language model learning from human feedback. If a single gradient update makes the model much more likely to generate certain patterns, the model might suddenly produce outputs that are completely off-distribution from its training, leading to reward model extrapolation errors and further degradation. The optimizer has no way to know it has moved too far until the damage is already done, and by that point, the policy may be in such a degraded state that subsequent updates cannot recover it without discarding all progress.