KL Divergence Penalty › KL Coefficient Selection

The KL coefficient $\beta$ is perhaps the most important hyperparameter in RLHF. It controls the basic tradeoff between two competing objectives that cannot be simultaneously maximized: learning from the reward signal as aggressively as possible, and staying close to the pre-trained distribution that is accumulated linguistic competence. There is no universally correct value of $\beta$; the right choice depends on the reward model quality, the magnitude of the desired behavior change, the training data distribution, and the response length distribution.

Think of $\beta$ as the stiffness of a spring connecting the policy to the reference model. A very stiff spring (high $\beta$) means the policy can only deviate a little from the reference before the restoring force becomes overwhelming. It will converge quickly because there is not much room to move, but it may never reach the optimal alignment behavior if that behavior requires substantial departure from the pre-trained distribution. A very weak spring (low $\beta$) allows the policy to explore widely but also means it can drift into regions of the space that are far from well-calibrated reward model territory, leading to reward hacking. The art of RLHF involves finding the spring stiffness that allows real movement toward better alignment without letting pathological exploitation of reward model errors.

The effects of the coefficient choice are asymmetric in their failure modes. Setting $\beta$ too low produces obvious failures: outputs that are repetitive, formulaic, or bizarre in ways that humans immediately notice. These failures are easy to detect through human evaluation and relatively easy to fix by increasing $\beta$. Setting $\beta$ too high produces subtle failures: the model barely moves from the reference, and improvements in alignment metrics are minimal or undetectable. This failure mode is harder to diagnose because the model still looks reasonable; it just has not learned much from the reward signal.

The key insight is that $\beta$ selection is not an abstract mathematical choice but an empirical one tied to the specific properties of your training setup. The optimal $\beta$ for a 7B parameter model may differ from that for a 70B parameter model; the optimal $\beta$ for a helpfulness-focused reward model may differ from that for a safety-focused one; the optimal $\beta$ for short-form responses may differ from that for long-form generation.

Understanding the qualitative effects of different coefficient ranges helps calibrate initial choices:

When $\beta$ is too low (below 0.01):

• The policy optimizes almost purely for reward, ignoring the reference constraint
• Susceptible to reward hacking and mode collapse
• May produce repetitive, formulaic outputs that exploit reward model quirks
• Loses diversity and coherence over time as the policy converges to a narrow mode of high-reward outputs

When $\beta$ is too high (above 1.0):

• Learning becomes extremely slow because the penalty overwhelms the reward signal
• The policy barely moves from the reference, even when the reward model gives clear guidance
• May never reach real alignment improvements
• Effectively wastes computational resources on training that produces minimal behavioral change

Within the typical range ($\beta \in [0.01, 0.2]$):

• Most successful RLHF implementations use values in this range
• InstructGPT used an initial value around 0.02
• Anthropic's Constitutional AI work used values around 0.001 to 0.01
• The optimal value depends on reward model quality and desired behavior change

When selecting a KL coefficient, consider these factors:

• Reward model confidence: If your reward model was trained on limited data or shows signs of miscalibration, use a higher $\beta$ to limit exploitation of its errors. A well-calibrated reward model trained on millions of preference comparisons can tolerate lower $\beta$ because there are fewer holes to exploit.

• Magnitude of desired behavior change: For small adjustments (such as reducing a specific narrow category of harmful outputs), lower $\beta$ suffices. For significant capability changes that require the model to learn substantially new behaviors, you may need to accept higher KL divergence and should verify that training produces behavioral changes rather than just optimizing the KL-penalized objective.

• Response length: Longer responses accumulate more KL. If you are optimizing for verbose outputs, the effective per-token penalty is spread over more tokens, so you might need to account for this in choosing $\beta$ or to normalize by length.

• Training stability: If you observe reward increasing while output quality degrades subjectively (which you can detect via human evaluation or automatic metrics), increase $\beta$ to strengthen the constraint. If training seems stable but the model is not improving, consider decreasing $\beta$ or increasing the reward model signal.