KL Divergence Penalty › Key Parameters

Understanding the key parameters helps practitioners set up RLHF training correctly and diagnose problems when they occur. These parameters are conceptual knobs that control the basic behavior of the alignment process, not incidental implementation details.

The parameters for the KL divergence implementation are:

• kl_coef ($\beta$): The weight of the KL penalty term. Controls the tradeoff between reward maximization and reference adherence. Higher values enforce stricter proximity to the reference at the cost of slower learning. Starting values typically range from 0.01 to 0.2 depending on the application.

• target_kl: The desired KL divergence value (in nats) for adaptive controllers. This is the setpoint for the feedback loop. A target of 6 nats is common in practice, corresponding to a moderate level of divergence that allows real behavioral change while preserving most of the reference model's capabilities. The right target depends on response length: a target of 6 nats spread over 100 tokens means an average of 0.06 nats per token, which is relatively small distributional shifts at each individual position.

• horizon: In the smooth adaptive controller, this determines the timescale over which adaptation occurs. Longer horizons make the controller less reactive to transient fluctuations but also slower to respond to real regime changes during training.