PPO Algorithm › The Complete PPO Objective

The full PPO objective combines three terms: policy improvement, value function training, and exploration. These components work together synergistically. Accurate value estimates enable meaningful advantages, while exploration discovers strategies that improve both the policy and value function. Training all three simultaneously with a shared network backbone allows the network to develop representations that serve all three purposes at once, improving sample efficiency compared to training separate networks.

The complete objective is:

$$L^{\text{PPO}}(\theta) = \mathbb{E}_t\left[L^{\text{CLIP}}(\theta) - c_1 L^{\text{VF}}(\theta) + c_2 S[\pi_\theta](s_t)\right]$$

Each component serves a distinct purpose in training a capable agent:

• $L^{\text{PPO}}(\theta)$: the complete PPO objective function, to be maximized
• $\mathbb{E}_t$: expectation over timesteps in the batch
• $L^{\text{CLIP}}(\theta)$: the clipped surrogate objective for policy improvement, defined earlier
• $c_1$: coefficient for the value function loss, typically 0.5
• $L^{\text{VF}}(\theta)$: the value function loss, which encourages accurate value estimates
• $c_2$: coefficient for the entropy bonus, typically 0.01
• $S[\pi_\theta](s_t)$: the entropy of the policy distribution at state $s_t$, which encourages exploration
• $\theta$: parameters of both the policy and value networks, when they share parameters
• $s_t$: the state at timestep $t$

The value function loss has a negative sign because we minimize loss while maximizing the overall objective. This sign convention means that minimizing value function error increases the overall objective, aligning all components toward the same optimization direction.

The three terms work together synergistically. The clipped surrogate loss $L^{\text{CLIP}}$ improves the policy by increasing probabilities of high-advantage actions while respecting the trust region constraint. The value function loss $L^{\text{VF}}$ trains the value function to make accurate predictions, which are needed for computing meaningful advantages in future updates. The entropy term $S$ encourages exploration by penalizing overly deterministic policies, preventing premature convergence and helping the agent discover better strategies.

The value function $V_\phi(s)$ predicts expected cumulative returns and is trained alongside the policy. Accurate value estimates are needed because the value function is the baseline in advantage computation. Without accuracy, advantages become noisy and unreliable, leading to high-variance policy gradients that destabilize training. Think of the value function as a sports commentator who tells you whether a play was good or bad: if the commentator has no idea what a typical play looks like, their assessments of "better than average" or "worse than average" are meaningless. The policy can only learn from advantage estimates if those estimates are grounded in an accurate model of expected future rewards.

The value function predicts expected cumulative future rewards starting from each state under the current policy. We train the value function by minimizing the mean squared error between its predictions and target values:

$$L^{\text{VF}}(\phi) = \mathbb{E}_t\left[(V_\phi(s_t) - V_t^{\text{target}})^2\right]$$

Understanding each component reveals the standard regression structure of value function training:

• $L^{\text{VF}}(\phi)$: the value function loss, the mean squared error
• $\mathbb{E}_t$: expectation over timesteps in the batch
• $\phi$: parameters of the value function network
• $V_\phi(s_t)$: the value function's prediction for state $s_t$ under current parameters
• $V_t^{\text{target}}$: the target value we want the value function to predict
• $s_t$: the state at timestep $t$
• $(V_\phi(s_t) - V_t^{\text{target}})^2$: the squared difference between prediction and target

The target value is computed as:

$$V_t^{\text{target}} = \hat{A}_t + V_{\phi_{\text{old}}}(s_t)$$

Each term in this target construction serves a specific purpose:

• $V_t^{\text{target}}$: the target value for the value function to predict at timestep $t$
• $\hat{A}_t$: the advantage estimate at timestep $t$, computed using GAE
• $V_{\phi_{\text{old}}}(s_t)$: the value prediction from the old value function, before this update
• $\phi_{\text{old}}$: parameters of the value function from the previous update
• $s_t$: the state at timestep $t$

This target construction guides the value function to predict returns accurately using computed advantages. The advantage $\hat{A}_t$ represents how much better actual returns were than expected, or equivalently, the residual between actual returns and predicted returns. Adding this residual to the old value estimate $V_{\phi_{\text{old}}}(s_t)$ yields an improved return estimate.

This bootstrapping approach allows the value function to learn from its own predictions while incorporating new information from observed rewards. The process is iterative: better value estimates lead to better advantage estimates, which lead to better policy updates, which generate better training data for the value function. Some implementations also clip the value function loss similarly to the policy loss, preventing large changes to the value function that might destabilize this iterative process. This value clipping is particularly useful in RLHF settings where the value function must adapt to reward model outputs that can change significantly during alignment training.

Entropy measures randomness in the policy's action distribution. High entropy spreads probability across multiple actions instead of putting on one choice. Encouraging higher entropy prevents the policy from becoming deterministic too early, which keeps exploration alive and enables discovery of better strategies. Without the entropy bonus, PPO tends to quickly commit to whatever actions seem best in the current training data, potentially missing superior alternatives that have not yet been explored. The entropy bonus is a small but important regularizer that maintains the policy's flexibility throughout training.

For a policy distribution at state $s_t$, entropy is defined as:

$$S[\pi_\theta](s_t) = -\sum_a \pi_\theta(a|s_t) \log \pi_\theta(a|s_t)$$

Each component of this formula connects to information-theoretic concepts:

• $S[\pi_\theta](s_t)$: the entropy of the policy distribution at state $s_t$
• $\sum_a$: sum over all possible actions in the action space
• $a$: an action from the action space
• $\pi_\theta(a|s_t)$: the probability of action $a$ in state $s_t$ under the current policy
• $\log \pi_\theta(a|s_t)$: the log probability of action $a$
• $\theta$: parameters of the policy network
• $s_t$: the state at timestep $t$

This entropy term encourages exploration by rewarding policies that maintain uncertainty over actions. The formula measures the expected information content, or surprise, in the policy's action distribution. Mathematically, when action probabilities are uniform, meaning all actions are equally likely, the sum of $\pi_\theta(a|s_t) \log \pi_\theta(a|s_t)$ terms is maximized in magnitude, giving high entropy. When the policy is deterministic, one action has probability 1 and all others have probability 0, so the entropy collapses to zero.

When all actions have similar probabilities, showing high uncertainty, entropy is maximized. This reflects the fact that an observer would be maximally uncertain about which action the policy will choose. When the policy becomes deterministic, meaning one action has probability near 1 and all others near 0, entropy approaches zero. The policy reveals little information because its behavior is predictable. While deterministic policies can be desirable in final deployment, during training we need exploration to discover better strategies.

The coefficient $c_2$ (typically 0.01) controls the exploration-exploitation tradeoff. Higher values encourage exploration but may slow convergence to optimal behavior. Lower values enable faster convergence but risk getting stuck in suboptimal local minima. In language model fine-tuning, the entropy coefficient is often set quite small because the base model already has a well-developed prior over language, and aggressive exploration would destroy linguistic coherence. The goal is gentle nudging toward human-preferred outputs, not wholesale behavioral reshaping.

The typical coefficient values that Schulman et al. used in the original paper are:

• $c_1 = 0.5$: weight for value function loss
• $c_2 = 0.01$: weight for entropy bonus
• $\epsilon = 0.2$: clipping parameter

Sharing parameters between policy and value networks, which is common in practice, combines all three losses for joint optimization. This parameter sharing encourages the network to learn representations useful for both predicting values and selecting actions. It often improves sample efficiency compared to separate networks. The shared layers act as a general-purpose feature extractor: features that help predict which actions lead to good outcomes (value function) are often the same features that should drive action selection (policy). The separate output heads then specialize these shared representations for their specific task.