PPO Algorithm › Worked Example: Tracing a Single PPO Update

Before implementing PPO in code, let's trace through one complete update step numerically. This example uses tiny numbers to keep the arithmetic tractable, but the procedure is identical for million-parameter networks.

Suppose we have collected a short trajectory with three timesteps. The states, actions, rewards, and value estimates are:

The episode terminates after timestep 2, so $V(s_3) = 0$. We use $\gamma = 0.99$ and $\lambda = 0.95$.

Step 1: Compute TD errors. Working forward through the trajectory:

$$\begin{aligned} \delta_0 &= r_0 + \gamma V(s_1) - V(s_0) = 1.0 + 0.99 \times 0.5 - 0.8 = 0.695 \\ \delta_1 &= r_1 + \gamma V(s_2) - V(s_1) = 0.0 + 0.99 \times 1.2 - 0.5 = 0.688 \\ \delta_2 &= r_2 + \gamma V(s_3) - V(s_2) = 2.0 + 0.99 \times 0.0 - 1.2 = 0.800 \end{aligned}$$

Step 2: Compute GAE advantages. Working backwards through the trajectory, initializing $\hat{A}_3 = 0$:

$$\begin{aligned} \hat{A}_2 &= \delta_2 + \gamma \lambda \hat{A}_3 = 0.800 + 0.99 \times 0.95 \times 0 = 0.800 \\ \hat{A}_1 &= \delta_1 + \gamma \lambda \hat{A}_2 = 0.688 + 0.99 \times 0.95 \times 0.800 = 0.688 + 0.752 = 1.440 \\ \hat{A}_0 &= \delta_0 + \gamma \lambda \hat{A}_1 = 0.695 + 0.99 \times 0.95 \times 1.440 = 0.695 + 1.354 = 2.049 \end{aligned}$$

The advantage at $t=0$ is largest because the trajectory begins with a low value prediction (0.8) but transitions to states with higher-than-expected cumulative reward.

Step 3: Normalize advantages. Standard PPO normalizes advantages to have zero mean and unit variance. The mean is $(2.049 + 1.440 + 0.800) / 3 = 1.430$. The variance is approximately $0.533$, giving standard deviation $\approx 0.730$. Normalizing:

$$\begin{aligned} \hat{A}_0^{\text{norm}} &= (2.049 - 1.430) / 0.730 = 0.848 \\ \hat{A}_1^{\text{norm}} &= (1.440 - 1.430) / 0.730 = 0.014 \\ \hat{A}_2^{\text{norm}} &= (0.800 - 1.430) / 0.730 = -0.863 \end{aligned}$$

Step 4: Compute probability ratios. After one gradient step, suppose the new policy has log-probabilities: $\log \pi_\theta(a_0 \mid s_0) = -0.511$, $\log \pi_\theta(a_1 \mid s_1) = -1.204$, $\log \pi_\theta(a_2 \mid s_2) = -0.693$.

$$\begin{aligned} r_0 &= \exp(-0.511 - (-0.693)) = \exp(0.182) = 1.20 \\ r_1 &= \exp(-1.204 - (-1.386)) = \exp(0.182) = 1.20 \\ r_2 &= \exp(-0.693 - (-0.405)) = \exp(-0.288) = 0.75 \end{aligned}$$

Step 5: Apply clipping with $\epsilon = 0.2$. The clip bounds are $[0.8, 1.2]$.

For $t=0$: $r_0 = 1.20$, $A_0^{\text{norm}} = 0.848$ (positive advantage). The unclipped term is $1.20 \times 0.848 = 1.018$. The clipped ratio is $\min(1.20, 1.20) = 1.20$, so the clipped term is also $1.20 \times 0.848 = 1.018$. Both terms equal, so $L_0 = 1.018$. Note that $r_0 = 1 + \epsilon$ exactly: the ratio is right at the boundary.

For $t=1$: $r_1 = 1.20$, $A_1^{\text{norm}} = 0.014$ (small positive). Similarly $L_1 = 1.20 \times 0.014 = 0.017$.

For $t=2$: $r_2 = 0.75$, $A_2^{\text{norm}} = -0.863$ (negative advantage). The unclipped term is $0.75 \times (-0.863) = -0.647$. The clipped ratio clips 0.75 to 0.8, giving clipped term $0.8 \times (-0.863) = -0.690$. The $\min(-0.647, -0.690) = -0.690$ (more negative wins). So $L_2 = -0.690$.

Here, $r_2 = 0.75 < 1 - \epsilon = 0.8$, so clipping activates and the clipped objective is flatter (less negative) than the unclipped one. This means the optimizer does not receive a gradient signal pushing $r_2$ below 0.8 further, preventing excessive suppression of action $a_2$.

Step 6: Average the objective. $L^{\text{CLIP}} = (1.018 + 0.017 + (-0.690)) / 3 = 0.115$.

This positive value indicates the gradient step improved the objective. On subsequent passes through the same data (multiple epochs), the ratios will update further, and clipping will activate when the policy has changed enough.