PPO Algorithm › The Probability Ratio

The probability ratio is central to PPO. It quantifies how much the new policy's probability for an action differs from the old policy's probability. By examining this ratio across all observed state-action pairs, you can assess whether the policy is changing appropriately or excessively. Think of the probability ratio as a sensitivity meter: a ratio of 1 means the policy is unchanged for this action, values greater than 1 mean the policy is becoming more likely to take this action, and values less than 1 mean the policy is becoming less likely. The ratio captures the entire story of how the policy is changing, action by action, state by state.

We define the ratio as:

Understanding what each symbol represents helps illuminate why this ratio captures policy change so effectively:

• $r_t(\theta)$: the importance sampling ratio at timestep $t$
• $\pi_\theta(a_t|s_t)$: probability of action $a_t$ in state $s_t$ under the new policy with parameters $\theta$
• $\pi_{\theta_{\text{old}}}(a_t|s_t)$: probability of the same action in the same state under the old policy
• $a_t$: the action taken at timestep $t$
• $s_t$: the state at timestep $t$
• $\theta$: parameters of the new policy
• $\theta_{\text{old}}$: parameters of the old policy

This ratio captures how much more or less likely action $a_t$ becomes under the new policy compared to the old one. The interpretation is intuitive and direct:

• $r_t(\theta) = 1$: the new policy assigns the same probability to this action; no change from the old policy
• $r_t(\theta) > 1$: the new policy makes this action more likely (for example, $r_t = 2$ means the new policy is twice as likely to take this action)
• $r_t(\theta) < 1$: the new policy makes this action less likely (for example, $r_t = 0.5$ means the new policy is half as likely to take this action)

This ratio directly measures behavioral change. A ratio of 1 everywhere indicates identical policies, while very large or very small ratios indicate dramatic behavioral shifts. This makes the ratio ideal for clipping to constrain policy changes. Importantly, the ratio is also the right quantity for importance sampling: when we use old data to evaluate a new policy, we multiply by the ratio to correct for the fact that we collected data under a different distribution. This dual role as both a measure of policy change and an importance weight makes the ratio the natural pivot point for the PPO objective.

The standard policy gradient objective can be rewritten using this ratio, revealing its basic role in policy optimization:

To understand how the ratio enables policy improvement, examine what each term contributes:

• $L^{\text{PG}}(\theta)$: the policy gradient objective function we want to maximize
• $\mathbb{E}_t$: expectation over timesteps in collected trajectories
• $r_t(\theta)$: the probability ratio at timestep $t$, equal to $\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$
• $A_t$: the advantage estimate at timestep $t$, measuring how much better the action taken was compared to the average action
• $\theta$: parameters of the policy being optimized

When $\theta = \theta_{\text{old}}$, we have $r_t(\theta) = 1$ everywhere, and the objective reduces to the simple advantage-weighted sum. The gradient of this objective at $\theta = \theta_{\text{old}}$ equals the standard policy gradient, confirming that this formulation is equivalent to what we derived earlier.

The problem becomes clear when we consider what happens as we optimize this objective. As the optimizer works to maximize the expected reward, the ratio can become arbitrarily large or small. If an action had positive advantage, showing it was better than expected, unconstrained optimization would keep increasing its probability without bound. The gradient always points toward making good actions more likely and bad actions less likely, but nothing in this formulation limits how far the policy can shift. This is precisely the instability problem that TRPO addressed with its KL constraint, and that PPO addresses with clipping.