PPO Algorithm › The Clipped Objective

PPO's key innovation is straightforward: clip the probability ratio to remove incentives for excessive policy changes. Rather than imposing a hard constraint requiring complex optimization machinery, PPO modifies the objective function itself, so large policy changes provide no additional benefit. The clipping mechanism is self-limiting: once the policy has changed enough to fall outside the allowed range, the gradient signal turns off, and the optimizer cannot push the policy further in that direction.

Think of the clipped objective as a reward system with diminishing returns for policy changes. The policy earns credit for adjusting action probabilities in the direction indicated by the advantage, but only up to a point. Beyond the clip threshold, additional changes earn no extra credit. This is like an employer who rewards employees for working overtime, but only up to a cap: you can work as many hours as you want, but you only get paid extra for the first few hours of overtime. The cap prevents extreme behavior without eliminating the incentive entirely.

The clipped surrogate objective is:

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[\min\left(r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t\right)\right]$$

where $\epsilon$ is a hyperparameter, typically set between 0.1 and 0.2, that defines the trust region width. This single parameter controls how much the policy can change in each update. Larger $\epsilon$ allows more aggressive updates. Smaller $\epsilon$ enforces tighter constraints and slower learning.

To understand precisely how clipping constrains the ratio, we need to examine the clip function itself. The clip function constrains the probability ratio to remain within the interval $[1-\epsilon, 1+\epsilon]$, defined as:

$$\text{clip}(r, 1-\epsilon, 1+\epsilon) = \begin{cases} 1-\epsilon & \text{if } r < 1-\epsilon \\ r & \text{if } 1-\epsilon \leq r \leq 1+\epsilon \\ 1+\epsilon & \text{if } r > 1+\epsilon \end{cases}$$

Each component of this piecewise function serves a specific purpose:

• $r$: the input value to be clipped; in PPO, this is $r_t(\theta)$, the probability ratio
• $1-\epsilon$: the lower bound of the allowed range
• $1+\epsilon$: the upper bound of the allowed range
• The function returns $r$ unchanged if it's within the bounds, otherwise returns the nearest bound

The key intuition is that this clipping removes the gradient signal when the ratio moves outside the trust region. Consider what happens during optimization: you want the optimizer to adjust parameters to increase the objective. If the new policy is already making an action much more likely, with a ratio greater than $1+\epsilon$, or much less likely, with a ratio less than $1-\epsilon$, than the old policy, clipping prevents the optimizer from pushing it even further in that direction. The clipped term becomes constant with respect to the parameters, meaning its gradient is zero, so there is no signal encouraging further movement in that direction.

This clipped ratio is then used in computing the clipped surrogate objective. Returning to the full $L^{\text{CLIP}}(\theta)$ formula and examining it in detail:

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[\min\left(r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t\right)\right]$$

Each component plays a specific role in creating stable policy updates:

• $L^{\text{CLIP}}(\theta)$: the clipped surrogate objective that PPO maximizes
• $\mathbb{E}_t$: expectation over all timesteps in the collected batch
• $\min(\cdot, \cdot)$: takes the smaller of the two arguments (the pessimistic bound)
• $r_t(\theta)$: the probability ratio $\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$
• $A_t$: the advantage estimate at timestep $t$
• $\epsilon$: the clipping parameter, typically 0.1 to 0.2, that defines the trust region
• $\text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)$: constrains the ratio to the interval $[1-\epsilon, 1+\epsilon]$

The $\min$ operation is important. It selects the smaller of the unclipped and clipped objectives. This keeps a pessimistic (conservative) bound on the improvement. By always choosing the lower value, we prevent the optimizer from being overly optimistic about improvements that would require large policy changes. We will analyze both cases based on the sign of the advantage.

Case 1: Positive Advantage ($A_t > 0$)

When an action is better than expected (positive advantage), we want to increase its probability. The unclipped objective's gradient pushes the policy to make this action more likely, but PPO limits the extent.

• If $r_t(\theta) > 1 + \epsilon$: The clipped term $(1+\epsilon) A_t$ is smaller than $r_t(\theta) A_t$, since $A_t > 0$ and $1+\epsilon < r_t(\theta)$. The $\min$ selects the clipped term $(1+\epsilon) A_t$, which is constant with respect to $\theta$, so the gradient is zero and provides no incentive for us to increase the ratio further.
• If $r_t(\theta) \leq 1 + \epsilon$: Both terms are equal or the unclipped term is smaller, so normal optimization proceeds.

This means that once the probability ratio exceeds $1+\epsilon$, the optimizer receives no additional reward for increasing it further. The policy has already been sufficiently encouraged to take this good action. Importantly, the policy can still take that action at a higher probability than the old policy, it just does not receive additional credit for pushing the probability even higher.

Case 2: Negative Advantage ($A_t < 0$)

When an action is worse than expected (negative advantage), we want to decrease its probability. The unclipped objective's gradient encourages this, but PPO limits the extent:

• If $r_t(\theta) < 1 - \epsilon$: Since $A_t < 0$, multiplying by smaller values makes the product more negative. The clipped term $(1-\epsilon) A_t$ is larger (less negative) than the unclipped term $r_t(\theta) A_t$ because $r_t(\theta) < 1-\epsilon$. The $\min$ selects the more negative unclipped term. Since the clipped term $(1-\epsilon) A_t$ is constant, its gradient is zero and the objective becomes flat once $r_t(\theta)$ drops below $1-\epsilon$, preventing the policy from decreasing the probability further.
• If $r_t(\theta) \geq 1 - \epsilon$: Normal optimization proceeds.

This prevents the policy from becoming too averse to actions that happened to have negative advantage. Such actions might still be valuable in other states, and excessively penalizing them could harm overall performance. The clipping protects both ends: it prevents overconfidence about good actions and overpunishment of bad ones.

The following figure illustrates this behavior: