PPO Algorithm: Proximal Policy Optimization for Stable RL

In the previous chapter, we explored policy gradient methods and saw how REINFORCE directly optimizes a policy by following the gradient of expected reward. While mathematically elegant, vanilla policy gradients are notoriously unstable during training. A single large gradient update can catastrophically degrade policy performance, and recovery is difficult. The algorithm might spend hundreds of update steps undoing damage from one reckless step. This instability prevented policy gradient methods from scaling to complex domains like language model fine-tuning, where each update step is expensive and evaluation is slow.

Proximal Policy Optimization (PPO), introduced by Schulman et al. in 2017, addresses this instability through a deceptively simple mechanism: it clips the objective function to prevent updates that change the policy too drastically. Rather than solving a complex constrained optimization problem, PPO modifies the reward signal itself so the optimizer cannot benefit from stepping too far from the current policy. The result is an algorithm that is simultaneously simpler to implement, more computationally efficient, and more stable than its predecessors.

Think of PPO as a careful hiker who always stays within a reliable map region. The hiker can see the general direction of higher elevation and wants to move toward it. But the hiker also knows that map accuracy degrades quickly outside the territory already explored. So the hiker commits to moving only a short distance with each step, updating the map from the new vantage point, and then deciding where to move next. The hiking speed is not zero, which would be no progress at all, but it is bounded, which prevents walking off a cliff based on a faulty extrapolation. PPO implements exactly this constraint: bound each update step so you remain in territory where your gradient estimates are trustworthy.

Understanding PPO is needed for anyone working with reinforcement learning from human feedback (RLHF) in language models. Modern alignment pipelines, including those behind ChatGPT, Claude, and Gemini, rely heavily on PPO or close variants to translate human preference data into updated model behavior. The algorithm's stability and implementation simplicity made it the natural choice when researchers needed to apply RL to language models with billions of parameters, where unstable training would be extraordinarily expensive to debug or recover from.

This chapter covers PPO in depth from the ground up. We start with the specific failure modes of unconstrained policy gradient optimization, move through the trust region methods that motivated PPO's design, then derive the clipped objective, work through Generalized Advantage Estimation, assemble the complete multi-component objective, and finally implement a working PPO agent in PyTorch. We include a numerical worked example so you can trace every computation by hand before running code. By the end, you will understand what PPO does, why each design choice exists, and what goes wrong when you deviate from it.

The instability in vanilla policy gradient methods is not a superficial implementation problem that can be fixed with a better learning rate schedule. It is a structural problem rooted in how the gradient signal relates to policy performance. To understand why PPO's solution is the right one, we need to be precise about exactly what goes wrong.

Recall that the policy gradient takes the form:

To understand why this formula creates practical difficulties, examine what each component contributes:

• $\nabla_\theta J(\theta)$: the gradient of the expected cumulative reward with respect to policy parameters, showing the direction to adjust parameters to increase expected reward
• $\mathbb{E}_{\tau \sim \pi_\theta}$: expectation over trajectories sampled from the current policy $\pi_\theta$
• $\theta$: parameters of the policy network that we are optimizing
• $J(\theta)$: the expected cumulative reward under policy $\pi_\theta$
• $\tau$: a trajectory (sequence of states and actions) sampled from the current policy
• $\pi_\theta(a_t | s_t)$: the probability of taking action $a_t$ in state $s_t$ under the current policy
• $T$: the time horizon, the length of the episode
• $A_t$: the advantage function at time $t$, estimating how much better action $a_t$ is compared to the average action in state $s_t$
• $\nabla_\theta \log \pi_\theta(a_t | s_t)$: the gradient of the log probability with respect to policy parameters, showing the direction to adjust parameters to make action $a_t$ more likely
• $\sum_{t=0}^{T}$: sum over all timesteps in the trajectory from 0 to T

The basic issue with this formulation is that the gradient provides no guidance about step size. The policy gradient theorem tells us which direction to move parameters to increase expected reward, but it remains entirely silent about how far we should move in that direction. This is analogous to knowing that walking north takes you closer to your destination but not knowing whether to take one step or one hundred steps. A step in the gradient direction improves performance locally, within an infinitesimally small neighborhood around the current parameters. However, nothing in the mathematics prevents taking such a large step that you overshoot into a region where the policy performs terribly. Policy performance is not smooth across the parameter space: a policy that seems promising can suddenly fail after a large update.

This problem is especially acute because of three interconnected challenges:

• Non-stationary data distribution: The policy generates its own training data. When the policy changes significantly, the state distribution also changes, potentially invalidating previously learned value estimates.
• High variance: Policy gradient estimates are inherently noisy, which makes it difficult to distinguish signal from noise.
• Irreversibility: A bad update might move the policy to a region where it never encounters states that would help it recover.

Consider a language model learning from human feedback. If a single gradient update makes the model much more likely to generate certain patterns, the model might suddenly produce outputs that are completely off-distribution from its training, leading to reward model extrapolation errors and further degradation. The optimizer has no way to know it has moved too far until the damage is already done, and by that point, the policy may be in such a degraded state that subsequent updates cannot recover it without discarding all progress.

The insight behind trust region methods is to constrain how much the policy can change in each update. Rather than blindly following the gradient wherever it leads, we optimize the policy subject to a constraint that keeps the new policy "close" to the old one. This approach recognizes a basic tension in optimization: we want to improve the policy as quickly as possible. However, our confidence in the improvement direction decreases the further we move from where we collected our data. Gradients computed from sampled trajectories accurately describe performance improvement near the current policy, but extrapolating far from observed data becomes unreliable.

Think of a trust region as the area on a topographic map that your GPS unit has surveyed at high resolution. Within that region, the contour lines are accurate and you can safely navigate. Outside that region, the map was generated from coarser satellite data and might have cliffs, valleys, or impassable terrain that looks navigable on the map. You would not plan a route that takes you far outside your high-resolution zone. Similarly, the policy gradient is an accurate local guide only within the neighborhood of states and actions you have recently observed. Venturing far outside that neighborhood means relying on gradient extrapolations that may be wildly wrong.

The key insight behind all trust region methods is that gradient estimates are only reliable near the data they were computed from. Moving the policy far from the current policy means the state-action distribution shifts, and the advantages computed from old data no longer accurately reflect the new policy's performance. The trust region constraint formalizes the boundary beyond which you should not venture without collecting new data. Enforcing this boundary ensures that the objective function improvement you see during optimization corresponds to actual performance improvement in the environment.

Trust Region Policy Optimization (TRPO), PPO's predecessor, formalizes this using KL divergence as a constraint. The KL divergence measures how different two probability distributions are, which makes it a natural choice for measuring policy similarity. Policies that assign similar probabilities to actions have low KL divergence, while policies that behave very differently have high KL divergence. This makes KL divergence a precise, mathematically grounded way to define "how much the policy has changed."

The TRPO objective addresses the policy update problem by maximizing expected advantage while explicitly constraining how much the policy can change. It maximizes the expected advantage weighted by the importance sampling ratio, which allows us to use data from the old policy to evaluate the new policy, while constraining the KL divergence between old and new policies:

To understand how this constrained optimization problem balances improvement against stability, examine each component:

• $\theta$: parameters of the new policy being optimized
• $\theta_{\text{old}}$: parameters of the old policy from which data was collected
• $\mathbb{E}_{s, a \sim \pi_{\theta_{\text{old}}}}$: expectation over states and actions sampled from trajectories collected using the old policy
• $s$: a state sampled from the state distribution under the old policy
• $a$: an action sampled from the old policy in state $s$
• $\pi_\theta(a|s)$: probability of action $a$ in state $s$ under the new policy
• $\pi_{\theta_{\text{old}}}(a|s)$: probability of action $a$ in state $s$ under the old policy
• $\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}$: the importance sampling ratio, which reweights data from the old policy to evaluate the new policy
• $A^{\pi_{\theta_{\text{old}}}}(s, a)$: advantage function computed under the old policy, measuring how much better action $a$ is than average in state $s$
• $\mathbb{E}_s$: expectation over states from the state distribution under the old policy
• $D_{\text{KL}}(\pi_{\theta_{\text{old}}} \| \pi_\theta)$: Kullback-Leibler divergence measuring how much the new policy distribution differs from the old policy distribution
• $\delta$: maximum allowed KL divergence, the trust region radius

This constraint ensures the new policy $\pi_\theta$ does not diverge too far from the old policy $\pi_{\theta_{\text{old}}}$ in terms of the probability distributions over actions. The ratio $\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}$ is called the importance sampling ratio and allows us to evaluate the new policy using data collected from the old policy. This ratio acts as a correction factor: if the new policy is twice as likely to take an action as the old policy, we weight that action's contribution twice as heavily to account for the fact that it would occur more frequently under the new policy.

TRPO guarantees monotonic improvement under certain conditions, meaning the policy never gets worse than the previous version. This guarantee provides the stability that vanilla policy gradients lack. However, the mathematical machinery required to enforce this guarantee comes with significant computational costs: computing Fisher information matrices and solving linear systems.

Solving the constrained optimization problem requires computing second-order derivatives, specifically the Fisher information matrix, and performing conjugate gradient optimization to solve a system of linear equations. This makes TRPO computationally expensive and difficult to implement correctly. The Fisher information matrix has size $n \times n$ where $n$ is the number of policy parameters, so even storing it is impractical for large neural networks. Computing the matrix-vector product required by conjugate gradient solvers requires special tricks involving second-order automatic differentiation. Computing the Fisher information matrix across multiple workers in distributed settings adds significant overhead.

PPO achieves similar stability guarantees with a first-order method by replacing the hard constraint with a penalty built directly into the objective function. Instead of solving a constrained optimization problem, PPO modifies the objective itself to discourage excessive policy changes. This transformation from constraint to penalty makes PPO dramatically simpler to implement while preserving the needed benefits of trust region methods. The resulting algorithm fits cleanly into any standard deep learning training loop, with no exotic optimization machinery required.

The probability ratio is central to PPO. It quantifies how much the new policy's probability for an action differs from the old policy's probability. By examining this ratio across all observed state-action pairs, you can assess whether the policy is changing appropriately or excessively. Think of the probability ratio as a sensitivity meter: a ratio of 1 means the policy is unchanged for this action, values greater than 1 mean the policy is becoming more likely to take this action, and values less than 1 mean the policy is becoming less likely. The ratio captures the entire story of how the policy is changing, action by action, state by state.

We define the ratio as:

Understanding what each symbol represents helps illuminate why this ratio captures policy change so effectively:

• $r_t(\theta)$: the importance sampling ratio at timestep $t$
• $\pi_\theta(a_t|s_t)$: probability of action $a_t$ in state $s_t$ under the new policy with parameters $\theta$
• $\pi_{\theta_{\text{old}}}(a_t|s_t)$: probability of the same action in the same state under the old policy
• $a_t$: the action taken at timestep $t$
• $s_t$: the state at timestep $t$
• $\theta$: parameters of the new policy
• $\theta_{\text{old}}$: parameters of the old policy

This ratio captures how much more or less likely action $a_t$ becomes under the new policy compared to the old one. The interpretation is intuitive and direct:

• $r_t(\theta) = 1$: the new policy assigns the same probability to this action; no change from the old policy
• $r_t(\theta) > 1$: the new policy makes this action more likely (for example, $r_t = 2$ means the new policy is twice as likely to take this action)
• $r_t(\theta) < 1$: the new policy makes this action less likely (for example, $r_t = 0.5$ means the new policy is half as likely to take this action)

This ratio directly measures behavioral change. A ratio of 1 everywhere indicates identical policies, while very large or very small ratios indicate dramatic behavioral shifts. This makes the ratio ideal for clipping to constrain policy changes. Importantly, the ratio is also the right quantity for importance sampling: when we use old data to evaluate a new policy, we multiply by the ratio to correct for the fact that we collected data under a different distribution. This dual role as both a measure of policy change and an importance weight makes the ratio the natural pivot point for the PPO objective.

The standard policy gradient objective can be rewritten using this ratio, revealing its basic role in policy optimization:

To understand how the ratio enables policy improvement, examine what each term contributes:

• $L^{\text{PG}}(\theta)$: the policy gradient objective function we want to maximize
• $\mathbb{E}_t$: expectation over timesteps in collected trajectories
• $r_t(\theta)$: the probability ratio at timestep $t$, equal to $\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$
• $A_t$: the advantage estimate at timestep $t$, measuring how much better the action taken was compared to the average action
• $\theta$: parameters of the policy being optimized

When $\theta = \theta_{\text{old}}$, we have $r_t(\theta) = 1$ everywhere, and the objective reduces to the simple advantage-weighted sum. The gradient of this objective at $\theta = \theta_{\text{old}}$ equals the standard policy gradient, confirming that this formulation is equivalent to what we derived earlier.

The problem becomes clear when we consider what happens as we optimize this objective. As the optimizer works to maximize the expected reward, the ratio can become arbitrarily large or small. If an action had positive advantage, showing it was better than expected, unconstrained optimization would keep increasing its probability without bound. The gradient always points toward making good actions more likely and bad actions less likely, but nothing in this formulation limits how far the policy can shift. This is precisely the instability problem that TRPO addressed with its KL constraint, and that PPO addresses with clipping.

PPO's key innovation is straightforward: clip the probability ratio to remove incentives for excessive policy changes. Rather than imposing a hard constraint requiring complex optimization machinery, PPO modifies the objective function itself, so large policy changes provide no additional benefit. The clipping mechanism is self-limiting: once the policy has changed enough to fall outside the allowed range, the gradient signal turns off, and the optimizer cannot push the policy further in that direction.

Think of the clipped objective as a reward system with diminishing returns for policy changes. The policy earns credit for adjusting action probabilities in the direction indicated by the advantage, but only up to a point. Beyond the clip threshold, additional changes earn no extra credit. This is like an employer who rewards employees for working overtime, but only up to a cap: you can work as many hours as you want, but you only get paid extra for the first few hours of overtime. The cap prevents extreme behavior without eliminating the incentive entirely.

The clipped surrogate objective is:

where $\epsilon$ is a hyperparameter, typically set between 0.1 and 0.2, that defines the trust region width. This single parameter controls how much the policy can change in each update. Larger $\epsilon$ allows more aggressive updates. Smaller $\epsilon$ enforces tighter constraints and slower learning.

To understand precisely how clipping constrains the ratio, we need to examine the clip function itself. The clip function constrains the probability ratio to remain within the interval $[1-\epsilon, 1+\epsilon]$, defined as:

Each component of this piecewise function serves a specific purpose:

• $r$: the input value to be clipped; in PPO, this is $r_t(\theta)$, the probability ratio
• $1-\epsilon$: the lower bound of the allowed range
• $1+\epsilon$: the upper bound of the allowed range
• The function returns $r$ unchanged if it's within the bounds, otherwise returns the nearest bound

The key intuition is that this clipping removes the gradient signal when the ratio moves outside the trust region. Consider what happens during optimization: you want the optimizer to adjust parameters to increase the objective. If the new policy is already making an action much more likely, with a ratio greater than $1+\epsilon$, or much less likely, with a ratio less than $1-\epsilon$, than the old policy, clipping prevents the optimizer from pushing it even further in that direction. The clipped term becomes constant with respect to the parameters, meaning its gradient is zero, so there is no signal encouraging further movement in that direction.

This clipped ratio is then used in computing the clipped surrogate objective. Returning to the full $L^{\text{CLIP}}(\theta)$ formula and examining it in detail:

Each component plays a specific role in creating stable policy updates:

• $L^{\text{CLIP}}(\theta)$: the clipped surrogate objective that PPO maximizes
• $\mathbb{E}_t$: expectation over all timesteps in the collected batch
• $\min(\cdot, \cdot)$: takes the smaller of the two arguments (the pessimistic bound)
• $r_t(\theta)$: the probability ratio $\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$
• $A_t$: the advantage estimate at timestep $t$
• $\epsilon$: the clipping parameter, typically 0.1 to 0.2, that defines the trust region
• $\text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)$: constrains the ratio to the interval $[1-\epsilon, 1+\epsilon]$

The $\min$ operation is important. It selects the smaller of the unclipped and clipped objectives. This keeps a pessimistic (conservative) bound on the improvement. By always choosing the lower value, we prevent the optimizer from being overly optimistic about improvements that would require large policy changes. We will analyze both cases based on the sign of the advantage.

Case 1: Positive Advantage ($A_t > 0$)

When an action is better than expected (positive advantage), we want to increase its probability. The unclipped objective's gradient pushes the policy to make this action more likely, but PPO limits the extent.

• If $r_t(\theta) > 1 + \epsilon$: The clipped term $(1+\epsilon) A_t$ is smaller than $r_t(\theta) A_t$, since $A_t > 0$ and $1+\epsilon < r_t(\theta)$. The $\min$ selects the clipped term $(1+\epsilon) A_t$, which is constant with respect to $\theta$, so the gradient is zero and provides no incentive for us to increase the ratio further.
• If $r_t(\theta) \leq 1 + \epsilon$: Both terms are equal or the unclipped term is smaller, so normal optimization proceeds.

This means that once the probability ratio exceeds $1+\epsilon$, the optimizer receives no additional reward for increasing it further. The policy has already been sufficiently encouraged to take this good action. Importantly, the policy can still take that action at a higher probability than the old policy, it just does not receive additional credit for pushing the probability even higher.

Case 2: Negative Advantage ($A_t < 0$)

When an action is worse than expected (negative advantage), we want to decrease its probability. The unclipped objective's gradient encourages this, but PPO limits the extent:

• If $r_t(\theta) < 1 - \epsilon$: Since $A_t < 0$, multiplying by smaller values makes the product more negative. The clipped term $(1-\epsilon) A_t$ is larger (less negative) than the unclipped term $r_t(\theta) A_t$ because $r_t(\theta) < 1-\epsilon$. The $\min$ selects the more negative unclipped term. Since the clipped term $(1-\epsilon) A_t$ is constant, its gradient is zero and the objective becomes flat once $r_t(\theta)$ drops below $1-\epsilon$, preventing the policy from decreasing the probability further.
• If $r_t(\theta) \geq 1 - \epsilon$: Normal optimization proceeds.

This prevents the policy from becoming too averse to actions that happened to have negative advantage. Such actions might still be valuable in other states, and excessively penalizing them could harm overall performance. The clipping protects both ends: it prevents overconfidence about good actions and overpunishment of bad ones.

The following figure illustrates this behavior:

PPO typically uses Generalized Advantage Estimation (GAE) to compute advantages with a controllable bias-variance tradeoff. The advantage function measures how much better an action is than average. True advantages depend on full trajectory information unavailable during learning, so we must estimate them from observed rewards. Different estimation approaches trade off bias against variance in different ways, and this tradeoff has a major impact on the stability and efficiency of training.

Think of advantage estimation as a forecasting problem. The one-step estimator is like a weather forecast that only looks at today's conditions: it is highly confident (low variance) but might be systematically wrong if today's conditions are atypical (high bias). The Monte Carlo estimator is like collecting many days of weather data before making a prediction: it captures complex patterns (low bias) but requires lots of data and is noisy (high variance). GAE is like a weighted ensemble of forecasts at different time horizons, combining the precision of short-term estimates with the accuracy of long-term ones.

A one-step estimate uses only the immediate reward and next state value. This provides low variance, since it depends on fewer random variables, but high bias, since it relies heavily on the accuracy of the value function. A Monte Carlo estimate uses all future rewards until episode end. This provides low bias, since it uses actual observed returns, but high variance, since it incorporates the randomness of many future actions and transitions. The quality of one-step estimates depends entirely on how good the value function is. If the value function is poor, one-step TD errors will be systematically wrong, and the policy will learn from corrupted signals.

GAE solves this estimation problem by taking an exponentially weighted average of temporal difference errors at different time horizons. The parameter $\lambda$ controls how quickly the weights decay as we look further into the future. This approach combines the benefits of using both short-term (low variance but high bias) and long-term (high variance but low bias) estimates. By tuning $\lambda$, we can find the sweet spot for our particular problem.

GAE is defined as:

Each component of this formula contributes to the bias-variance tradeoff:

• $\hat{A}_t^{\text{GAE}(\gamma, \lambda)}$: the GAE advantage estimate at timestep $t$
• $\sum_{l=0}^{\infty}$: sum over all future timesteps from the current timestep forward (in practice, truncated at episode end)
• $l$: the lookahead index, showing how many steps into the future we're considering
• $\gamma$: the discount factor, typically 0.99, which determines how much we value future rewards
• $\lambda$: the GAE parameter, typically 0.95, which controls the bias-variance tradeoff
• $(\gamma \lambda)^l$: the exponentially decaying weight for the $l$-step temporal difference error
• $\delta_{t+l}$: the temporal difference (TD) error at timestep $t+l$
• $t$: the current timestep

The temporal difference error, which is the building block for GAE, measures the discrepancy between what we expected and what we observed. It is defined as:

Understanding each term clarifies why TD errors are useful for advantage estimation:

• $\delta_t$: the temporal difference error at timestep $t$, measuring the difference between the observed reward plus next state value versus the current state value
• $r_t$: the immediate reward received at timestep $t$
• $\gamma$: the discount factor
• $V(s_t)$: the value function estimate for state $s_t$ (the predicted cumulative future reward)
• $V(s_{t+1})$: the value function estimate for the next state
• $\gamma V(s_{t+1})$: the discounted value of the next state
• $s_t$: the state at timestep $t$
• $s_{t+1}$: the next state at timestep $t+1$

The TD error intuition is clear: if our value function were perfect, the expected TD error would be zero. The value of the current state should equal the immediate reward plus the discounted value of the next state. A positive TD error indicates we received more reward than expected, suggesting the action was good. A negative TD error indicates we received less than expected. By accumulating these errors over multiple timesteps with exponentially decaying weights, GAE builds a richer signal that accounts for delayed consequences while limiting how much noise accumulates.

The tradeoff controlled by $\lambda$ determines how we combine information across time horizons:

• $\lambda = 0$ uses only one-step TD error (low variance, high bias)
• $\lambda = 1$ uses full Monte Carlo returns (high variance, low bias)

In practice, $\lambda = 0.95$ provides a good balance for most problems, weighting nearby TD errors heavily while still incorporating longer-term information with diminishing weight. For language models, this setting helps because the value function approximation is imperfect given the enormous state space, and GAE mitigates the impact of value function errors on advantage quality.

The recursive formulation for efficient computation eliminates the need to store and sum all future TD errors explicitly:

Each component of this recursive formula has a clear interpretation:

• $\hat{A}_t$: the advantage estimate at timestep $t$, computed recursively
• $\delta_t$: the temporal difference error at timestep $t$, equal to $r_t + \gamma V(s_{t+1}) - V(s_t)$
• $\gamma$: discount factor
• $\lambda$: GAE parameter
• $\hat{A}_{t+1}$: the advantage estimate for the next timestep, computed first in backward iteration
• $t$: the current timestep

The boundary condition is $\hat{A}_T = 0$ at the terminal timestep, since there are no future advantages after the episode ends. This recursive formulation is computationally efficient because we can compute all advantages in a single backward pass through the trajectory, starting from the end and working toward the beginning. Each computation reuses the result from the next timestep, avoiding redundant calculations. The algorithm is simple: iterate backwards, accumulating a running GAE value with decay, and store the result at each timestep.

Before implementing PPO in code, let's trace through one complete update step numerically. This example uses tiny numbers to keep the arithmetic tractable, but the procedure is identical for million-parameter networks.

Suppose we have collected a short trajectory with three timesteps. The states, actions, rewards, and value estimates are:

The episode terminates after timestep 2, so $V(s_3) = 0$. We use $\gamma = 0.99$ and $\lambda = 0.95$.

Step 1: Compute TD errors. Working forward through the trajectory:

Step 2: Compute GAE advantages. Working backwards through the trajectory, initializing $\hat{A}_3 = 0$:

The advantage at $t=0$ is largest because the trajectory begins with a low value prediction (0.8) but transitions to states with higher-than-expected cumulative reward.

Step 3: Normalize advantages. Standard PPO normalizes advantages to have zero mean and unit variance. The mean is $(2.049 + 1.440 + 0.800) / 3 = 1.430$. The variance is approximately $0.533$, giving standard deviation $\approx 0.730$. Normalizing:

Step 4: Compute probability ratios. After one gradient step, suppose the new policy has log-probabilities: $\log \pi_\theta(a_0 \mid s_0) = -0.511$, $\log \pi_\theta(a_1 \mid s_1) = -1.204$, $\log \pi_\theta(a_2 \mid s_2) = -0.693$.

Step 5: Apply clipping with $\epsilon = 0.2$. The clip bounds are $[0.8, 1.2]$.

For $t=0$: $r_0 = 1.20$, $A_0^{\text{norm}} = 0.848$ (positive advantage). The unclipped term is $1.20 \times 0.848 = 1.018$. The clipped ratio is $\min(1.20, 1.20) = 1.20$, so the clipped term is also $1.20 \times 0.848 = 1.018$. Both terms equal, so $L_0 = 1.018$. Note that $r_0 = 1 + \epsilon$ exactly: the ratio is right at the boundary.

For $t=1$: $r_1 = 1.20$, $A_1^{\text{norm}} = 0.014$ (small positive). Similarly $L_1 = 1.20 \times 0.014 = 0.017$.

For $t=2$: $r_2 = 0.75$, $A_2^{\text{norm}} = -0.863$ (negative advantage). The unclipped term is $0.75 \times (-0.863) = -0.647$. The clipped ratio clips 0.75 to 0.8, giving clipped term $0.8 \times (-0.863) = -0.690$. The $\min(-0.647, -0.690) = -0.690$ (more negative wins). So $L_2 = -0.690$.

Here, $r_2 = 0.75 < 1 - \epsilon = 0.8$, so clipping activates and the clipped objective is flatter (less negative) than the unclipped one. This means the optimizer does not receive a gradient signal pushing $r_2$ below 0.8 further, preventing excessive suppression of action $a_2$.

Step 6: Average the objective. $L^{\text{CLIP}} = (1.018 + 0.017 + (-0.690)) / 3 = 0.115$.

This positive value indicates the gradient step improved the objective. On subsequent passes through the same data (multiple epochs), the ratios will update further, and clipping will activate when the policy has changed enough.

The full PPO objective combines three terms: policy improvement, value function training, and exploration. These components work together synergistically. Accurate value estimates enable meaningful advantages, while exploration discovers strategies that improve both the policy and value function. Training all three simultaneously with a shared network backbone allows the network to develop representations that serve all three purposes at once, improving sample efficiency compared to training separate networks.

The complete objective is:

Each component serves a distinct purpose in training a capable agent:

• $L^{\text{PPO}}(\theta)$: the complete PPO objective function, to be maximized
• $\mathbb{E}_t$: expectation over timesteps in the batch
• $L^{\text{CLIP}}(\theta)$: the clipped surrogate objective for policy improvement, defined earlier
• $c_1$: coefficient for the value function loss, typically 0.5
• $L^{\text{VF}}(\theta)$: the value function loss, which encourages accurate value estimates
• $c_2$: coefficient for the entropy bonus, typically 0.01
• $S[\pi_\theta](s_t)$: the entropy of the policy distribution at state $s_t$, which encourages exploration
• $\theta$: parameters of both the policy and value networks, when they share parameters
• $s_t$: the state at timestep $t$

The value function loss has a negative sign because we minimize loss while maximizing the overall objective. This sign convention means that minimizing value function error increases the overall objective, aligning all components toward the same optimization direction.

The three terms work together synergistically. The clipped surrogate loss $L^{\text{CLIP}}$ improves the policy by increasing probabilities of high-advantage actions while respecting the trust region constraint. The value function loss $L^{\text{VF}}$ trains the value function to make accurate predictions, which are needed for computing meaningful advantages in future updates. The entropy term $S$ encourages exploration by penalizing overly deterministic policies, preventing premature convergence and helping the agent discover better strategies.

The value function $V_\phi(s)$ predicts expected cumulative returns and is trained alongside the policy. Accurate value estimates are needed because the value function is the baseline in advantage computation. Without accuracy, advantages become noisy and unreliable, leading to high-variance policy gradients that destabilize training. Think of the value function as a sports commentator who tells you whether a play was good or bad: if the commentator has no idea what a typical play looks like, their assessments of "better than average" or "worse than average" are meaningless. The policy can only learn from advantage estimates if those estimates are grounded in an accurate model of expected future rewards.

The value function predicts expected cumulative future rewards starting from each state under the current policy. We train the value function by minimizing the mean squared error between its predictions and target values:

Understanding each component reveals the standard regression structure of value function training:

• $L^{\text{VF}}(\phi)$: the value function loss, the mean squared error
• $\mathbb{E}_t$: expectation over timesteps in the batch
• $\phi$: parameters of the value function network
• $V_\phi(s_t)$: the value function's prediction for state $s_t$ under current parameters
• $V_t^{\text{target}}$: the target value we want the value function to predict
• $s_t$: the state at timestep $t$
• $(V_\phi(s_t) - V_t^{\text{target}})^2$: the squared difference between prediction and target

The target value is computed as:

Each term in this target construction serves a specific purpose:

• $V_t^{\text{target}}$: the target value for the value function to predict at timestep $t$
• $\hat{A}_t$: the advantage estimate at timestep $t$, computed using GAE
• $V_{\phi_{\text{old}}}(s_t)$: the value prediction from the old value function, before this update
• $\phi_{\text{old}}$: parameters of the value function from the previous update
• $s_t$: the state at timestep $t$

This target construction guides the value function to predict returns accurately using computed advantages. The advantage $\hat{A}_t$ represents how much better actual returns were than expected, or equivalently, the residual between actual returns and predicted returns. Adding this residual to the old value estimate $V_{\phi_{\text{old}}}(s_t)$ yields an improved return estimate.

This bootstrapping approach allows the value function to learn from its own predictions while incorporating new information from observed rewards. The process is iterative: better value estimates lead to better advantage estimates, which lead to better policy updates, which generate better training data for the value function. Some implementations also clip the value function loss similarly to the policy loss, preventing large changes to the value function that might destabilize this iterative process. This value clipping is particularly useful in RLHF settings where the value function must adapt to reward model outputs that can change significantly during alignment training.

Entropy measures randomness in the policy's action distribution. High entropy spreads probability across multiple actions instead of putting on one choice. Encouraging higher entropy prevents the policy from becoming deterministic too early, which keeps exploration alive and enables discovery of better strategies. Without the entropy bonus, PPO tends to quickly commit to whatever actions seem best in the current training data, potentially missing superior alternatives that have not yet been explored. The entropy bonus is a small but important regularizer that maintains the policy's flexibility throughout training.

For a policy distribution at state $s_t$, entropy is defined as:

Each component of this formula connects to information-theoretic concepts:

• $S[\pi_\theta](s_t)$: the entropy of the policy distribution at state $s_t$
• $\sum_a$: sum over all possible actions in the action space
• $a$: an action from the action space
• $\pi_\theta(a|s_t)$: the probability of action $a$ in state $s_t$ under the current policy
• $\log \pi_\theta(a|s_t)$: the log probability of action $a$
• $\theta$: parameters of the policy network
• $s_t$: the state at timestep $t$

This entropy term encourages exploration by rewarding policies that maintain uncertainty over actions. The formula measures the expected information content, or surprise, in the policy's action distribution. Mathematically, when action probabilities are uniform, meaning all actions are equally likely, the sum of $\pi_\theta(a|s_t) \log \pi_\theta(a|s_t)$ terms is maximized in magnitude, giving high entropy. When the policy is deterministic, one action has probability 1 and all others have probability 0, so the entropy collapses to zero.

When all actions have similar probabilities, showing high uncertainty, entropy is maximized. This reflects the fact that an observer would be maximally uncertain about which action the policy will choose. When the policy becomes deterministic, meaning one action has probability near 1 and all others near 0, entropy approaches zero. The policy reveals little information because its behavior is predictable. While deterministic policies can be desirable in final deployment, during training we need exploration to discover better strategies.

The coefficient $c_2$ (typically 0.01) controls the exploration-exploitation tradeoff. Higher values encourage exploration but may slow convergence to optimal behavior. Lower values enable faster convergence but risk getting stuck in suboptimal local minima. In language model fine-tuning, the entropy coefficient is often set quite small because the base model already has a well-developed prior over language, and aggressive exploration would destroy linguistic coherence. The goal is gentle nudging toward human-preferred outputs, not wholesale behavioral reshaping.

The typical coefficient values that Schulman et al. used in the original paper are:

• $c_1 = 0.5$: weight for value function loss
• $c_2 = 0.01$: weight for entropy bonus
• $\epsilon = 0.2$: clipping parameter

Sharing parameters between policy and value networks, which is common in practice, combines all three losses for joint optimization. This parameter sharing encourages the network to learn representations useful for both predicting values and selecting actions. It often improves sample efficiency compared to separate networks. The shared layers act as a general-purpose feature extractor: features that help predict which actions lead to good outcomes (value function) are often the same features that should drive action selection (policy). The separate output heads then specialize these shared representations for their specific task.

We will implement PPO for continuous control using a simple environment. This lets you focus on the algorithm rather than domain-specific complexity. The environment we use, Pendulum-v1 from OpenAI Gymnasium, requires a continuous torque input to balance an inverted pendulum. It is simple enough to train in minutes but complex enough to require real policy learning.

The actor-critic architecture is central to PPO. The actor is the policy: it takes a state and outputs a distribution over actions. The critic is the value function: it takes a state and outputs an estimate of the expected cumulative reward from that state. Sharing the lower layers between actor and critic allows both to benefit from the same learned representation of the environment, reducing the total number of parameters and speeding up learning.

This architecture implements the actor-critic pattern for continuous control. The shared feature extractor, two 64-unit hidden layers with tanh activations, learns representations useful for both policy and value prediction. The actor head outputs Gaussian parameters (mean and learnable log standard deviation), letting sampling of continuous actions with exploration noise. The standard deviation starts at 1.0 (since the log standard deviation is initialized to 0) and is learned independently of the mean, letting the policy to adjust its exploration width as training progresses. The critic head predicts state values for advantage computation. The get_action method samples actions during rollout collection. The evaluate_actions method computes log probabilities and entropy during policy updates, which are used to compute the probability ratios and entropy bonus in the loss function.

PPO requires collecting a batch of experience before updating the policy. The experience buffer stores all transitions from the current rollout, then computes advantages in a single backward pass before minibatch optimization begins. This separation of collection and optimization is basic to PPO's design: you collect data under the old policy, compute advantages, then optimize for multiple epochs, discarding the data when it becomes too stale.

The RolloutBuffer manages experience collection and advantage computation. The add method stores each transition (state, action, reward, value estimate, log probability, done flag) during rollout. The compute_returns_and_advantages method implements GAE by iterating backwards through the trajectory, computing temporal difference errors and accumulating them with exponential decay controlled by gamma and lambda. Note how episode boundaries are handled: when dones[t] is true, the bootstrapped next-state value is zero (the episode has ended), so the delta uses only the reward minus the current value. This backward pass efficiently computes advantages for all timesteps in a single sweep. The get_batches method shuffles the data and yields minibatches for multiple optimization epochs, improving sample efficiency by making multiple gradient updates per rollout.

The core PPO update is where all the mathematics we have discussed comes together. In each epoch, we iterate through the collected data in randomized minibatches, compute the three components of the PPO objective, combine them with their respective coefficients, and perform a gradient step. The key detail is computing ratios in log space for numerical stability: instead of computing $\pi_\theta(a) / \pi_{\text{old}}(a)$ directly (which can involve extremely small or large numbers), we compute $\exp(\log \pi_\theta(a) - \log \pi_{\text{old}}(a))$, which avoids floating point underflow or overflow.

Notice several important details in this implementation:

• Advantage normalization (zero mean, unit variance) stabilizes training by preventing advantages from dominating or being dominated by their scale. This is a necessary practical detail that the original paper does not emphasize but every successful implementation includes.
• Probability ratios are computed in log space for numerical stability: $r = \exp(\log \pi_{\theta}(a|s) - \log \pi_{\theta_{\text{old}}}(a|s))$, which avoids underflow when probabilities are very small.
• Gradient clipping at norm 0.5 prevents exploding gradients. This provides a secondary stability mechanism beyond the clipped objective.
• Multiple epochs of updates reuse collected data, improving sample efficiency. The data becomes stale over multiple epochs (the ratios drift away from 1), but the clipping mechanism limits how far the policy can drift within any single epoch.

The training loop orchestrates PPO by repeating the collect-compute-update cycle. Understanding this cycle is important: PPO is an on-policy algorithm at the level of rollouts (each rollout must come from the current policy), but it is off-policy at the level of gradient steps within a rollout (after the rollout, we run multiple gradient steps using the same data). This hybrid approach is what gives PPO its sample efficiency advantage over pure on-policy methods like REINFORCE.

The training results show the behavior of a deliberately compact 30,000-timestep run. The total episode count shows how many complete episodes occurred during the budget. The final average reward, computed over the last 10 episodes, indicates current policy performance, while the best average reward records the strongest short-window result. This small run is sufficient to demonstrate the PPO pipeline and its noisy learning dynamics, but it should not be read as a fully converged Pendulum solution.

Plotting training curves lets us verify that the algorithm is behaving correctly. We expect to see rewards improving over time, with high variance in individual episodes smoothing into a clear upward trend when averaged. Policy loss should initially be volatile as the policy learns rapidly, then stabilize. Value loss should decline as the value function learns to predict returns accurately.

PPO's performance depends on several key hyperparameters, and understanding what each one controls is needed for successful application. These hyperparameters interact with each other, so tuning one often requires adjusting others. The values below represent good starting points, but every domain benefits from some tuning.

The clipping parameter $\epsilon$ is the most basic hyperparameter because it directly controls the trust region. Smaller values make PPO more conservative, potentially requiring more rollouts to achieve the same policy improvement. Larger values allow faster learning but increase the risk of instability. The number of update epochs per rollout determines how much you squeeze out of each collected batch. More epochs improve sample efficiency (less environment interaction needed) but increase the risk that the policy drifts too far from the old policy within a single rollout. PPO's clipping limits this drift, but with many epochs and aggressive learning rates, the policy can still move substantially.

The key hyperparameters and their typical values are:

• Clip parameter ($\epsilon$): Controls policy change by defining the trust region as $[1-\epsilon, 1+\epsilon]$ for the probability ratio. Values of 0.1 to 0.3 are typical. Smaller values (e.g., 0.05) constrain updates too tightly and slow learning. Larger values (e.g., 0.5) provide insufficient constraint and risk instability.
• GAE lambda ($\lambda$): Bias-variance tradeoff for advantage estimation. Values of 0.95 to 0.99 are typical.
• Number of epochs: Number of passes through collected data. Using 3 to 10 epochs balances sample efficiency against overfitting to old data.
• Minibatch size: Larger batches provide more stable gradients but may overfit. Sizes of 32 to 256 are common.
• Rollout length: How much data to collect before updating. Longer rollouts provide better advantage estimates but slower iteration.
• Learning rate: Values of 1e-4 to 3e-4 are typical for PPO. You can use learning rate scheduling to improve convergence.

Language model implementations often require adjusted values. The next chapter explores LLM-specific considerations, including how to adapt the reward signal from a reward model, manage the KL penalty against the reference model, and handle the much larger action spaces of language generation.