RLHF Pipeline › RLHF Debugging
Language AI HandbookAll chapters RLHF Pipeline › RLHF DebuggingOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
RLHF training is notoriously difficult to debug. The interplay between the policy, reward model, and KL constraint creates many potential failure modes. Unlike standard supervised learning, where a rising validation loss is a clear and unambiguous signal that something is wrong, RLHF metrics can be simultaneously misleading in several directions at once: reward can increase while response quality decreases, KL divergence can remain controlled while mode collapse is developing, and entropy can look healthy while the policy is creating repetitive variations on a narrow set of response templates.
Effective debugging requires combining quantitative metrics with qualitative evaluation. No set of numerical metrics can substitute for regularly reading the model's outputs during training. A reward model score of 2.5 could represent excellent responses, or it could represent responses the policy has learned to craft specifically to exploit the reward model's scoring tendencies. The only way to tell the difference is to look at the text.
Key Metrics to Monitor
Effective RLHF debugging requires tracking multiple metrics throughout training:
import numpy as np
class RLHFMetricsTracker:
"""Track key metrics for RLHF debugging."""
def __init__(self):
self.metrics_history = {
"reward_mean": [],
"reward_std": [],
"kl_divergence": [],
"policy_loss": [],
"entropy": [],
"clip_fraction": [],
"response_length": [],
"unique_tokens": [],
}
def log_step(self, metrics_dict):
for key, value in metrics_dict.items():
if key in self.metrics_history:
self.metrics_history[key].append(value)
def check_health(self):
"""Check for common RLHF failure modes."""
warnings = []
if len(self.metrics_history["kl_divergence"]) > 100:
recent_kl = np.mean(self.metrics_history["kl_divergence"][-100:])
if recent_kl > 10:
warnings.append(
f"HIGH KL DIVERGENCE ({recent_kl:.2f}): Policy diverging from reference"
)
elif recent_kl < 0.01:
warnings.append(
f"LOW KL DIVERGENCE ({recent_kl:.4f}): Policy not learning"
)
if len(self.metrics_history["reward_mean"]) > 100:
recent_reward = np.mean(self.metrics_history["reward_mean"][-100:])
if recent_reward > 5:
warnings.append(
f"VERY HIGH REWARD ({recent_reward:.2f}): Possible reward hacking"
)
if len(self.metrics_history["entropy"]) > 100:
recent_entropy = np.mean(self.metrics_history["entropy"][-100:])
if recent_entropy < 0.1:
warnings.append(
f"LOW ENTROPY ({recent_entropy:.4f}): Policy becoming deterministic"
)
if len(self.metrics_history["response_length"]) > 100:
recent_len = np.mean(self.metrics_history["response_length"][-100:])
early_len = np.mean(self.metrics_history["response_length"][:100])
if recent_len < early_len * 0.5:
warnings.append(
"RESPONSE LENGTH COLLAPSED: Model creating very short responses"
)
elif recent_len > early_len * 2:
warnings.append(
"RESPONSE LENGTH EXPLOSION: Model creating very long responses"
)
return warningsimport numpy as np
tracker = RLHFMetricsTracker()
for i in range(200):
kl = 0.5 + 0.1 * np.random.randn() + i * 0.08 # Gradually increasing KL
tracker.log_step(
{
"kl_divergence": kl,
"reward_mean": 2.0 + 0.5 * np.random.randn() + i * 0.04,
"entropy": max(0.05, 1.0 - i * 0.004 + 0.1 * np.random.randn()),
"response_length": 100 + 10 * np.random.randn() - i * 0.2,
}
)
warnings = tracker.check_health()Health Check Results: ⚠️ HIGH KL DIVERGENCE (12.45): Policy diverging from reference ⚠️ VERY HIGH REWARD (7.95): Possible reward hacking
The metrics tracker successfully identifies the simulated anomalies. By monitoring the KL divergence and reward statistics we can catch issues like the "High KL" spike and "High Reward" events (simulating reward hacking) before they destabilize the entire training run.
Understanding what each metric tells you is as important as tracking it. KL divergence measures how far the current policy has moved from the reference model in terms of its probability distributions over responses. High KL is not inherently bad if reward has improved correspondingly, but high KL with stagnant or declining response quality is a warning sign that the policy has drifted into a region where the reward model is no longer a reliable proxy for human preferences. Entropy measures how deterministic the policy has become. Entropy collapsing to near zero while training is still ongoing suggests mode collapse is developing, even if reward appears stable.
The clip fraction (the fraction of PPO ratio updates that were clipped) is a particularly useful diagnostic. Values consistently above 0.3 indicate that the policy is changing faster than the clipped objective can accommodate, which means the old log-probabilities stored from the rollout phase are no longer accurate descriptions of the current policy. This is a sign that the PPO epoch count (how many times you update the policy on the same set of rollouts) is too high, or the learning rate is too large. Reducing either parameter should bring the clip fraction back to a healthier range of 0.05 to 0.2.
Common Failure Modes
Understanding these failure modes helps you diagnose and fix training issues:
Reward Hacking occurs when the policy finds exploits in the reward model that don't correspond to quality improvements. Signs include rapidly increasing reward with degrading response quality, or unusual patterns like excessive repetition or specific phrases.
import numpy as np
def detect_reward_hacking(responses, rewards, threshold_percentile=95):
"""Detect potential reward hacking by examining high-reward responses."""
threshold = np.percentile(rewards, threshold_percentile)
high_reward_indices = np.where(np.array(rewards) > threshold)[0]
flags = []
for idx in high_reward_indices:
response = responses[idx]
words = response.split()
if len(words) > 0:
unique_ratio = len(set(words)) / len(words)
if unique_ratio < 0.3:
flags.append(("repetition", idx, response[:100]))
if len(response) > 2000:
flags.append(("excessive_length", idx, f"Length: {len(response)}"))
elif len(response) < 10:
flags.append(("too_short", idx, response))
return flagsReward hacking is insidious because the numerical metrics look good while the model's behavior deteriorates. The reward model has a finite capacity and has only seen a finite amount of training data. The PPO policy, optimizing aggressively, can discover input patterns that the reward model was never trained to reject. Classic examples include: generating responses that begin with phrases the reward model associates with high quality (such as "Certainly! Here's a complete answer...") regardless of whether the content is helpful; creating responses of a specific length that correlates with high reward in the training data without giving proportional value; and inserting specific keywords that the reward model's training data associated with expert responses. Each of these behaviors increases the reward model score without making the response more useful.
Mode Collapse happens when the policy converges to creating nearly identical responses regardless of the prompt. Monitor response diversity and entropy throughout training.
import numpy as np
def compute_response_diversity(responses, n_samples=100):
"""Measure diversity of generated responses."""
unique_responses = len(set(responses[:n_samples]))
uniqueness_ratio = unique_responses / min(n_samples, len(responses))
all_words = []
for response in responses[:n_samples]:
all_words.extend(response.lower().split())
if len(all_words) > 0:
vocab_size = len(set(all_words))
type_token_ratio = vocab_size / len(all_words)
else:
type_token_ratio = 0
return {
"uniqueness_ratio": uniqueness_ratio,
"type_token_ratio": type_token_ratio,
"avg_response_length": np.mean([len(r) for r in responses[:n_samples]]),
}KL Explosion indicates the policy is moving too fast away from the reference model. This often precedes training instability:
Debugging Workflow
When RLHF training goes wrong, follow this systematic debugging approach:
-
Check the reward model first: Generate samples and manually verify that reward model scores align with your quality intuitions. A miscalibrated or overfitted reward model dooms PPO from the start.
-
Examine generated samples: Look at actual model outputs throughout training. Metrics can hide problems that become obvious when reading responses.
-
Verify the KL penalty is working: The policy should stay reasonably close to the reference. If responses look completely different from SFT outputs, the KL constraint may be too weak.
-
Monitor multiple metrics together: Single metrics can be misleading. High reward with low diversity suggests reward hacking. Low KL with no reward improvement suggests the policy isn't learning.
A systematic debugging session for a troubled RLHF run typically looks like this: first, freeze the policy and run the reward model on a held-out set of manually graded responses to check that its rankings match your intuitions. If the reward model consistently assigns higher scores to responses you would consider worse, the problem is in stage 2, and you need to retrain the reward model. If the reward model scores make sense but the policy is not improving, the issue is more likely in the PPO hyperparameters, such as too high a KL coefficient preventing meaningful updates, or too few rollout steps per iteration giving the advantage estimator high variance. If the policy is improving according to the reward model but human evaluation shows no improvement, the reward model is overfitting and its scores have become decoupled from actual quality.
import torch
def debug_rlhf_step(
policy_model, ref_model, reward_model, sample_prompts, tokenizer, device
):
"""Comprehensive debugging for a single RLHF step."""
debug_info = {}
policy_model.eval()
ref_model.eval()
with torch.no_grad():
responses = []
for prompt in sample_prompts[:5]:
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)
output_ids = policy_model.generate(
input_ids, max_new_tokens=100, do_sample=True, temperature=0.7
)
response = tokenizer.decode(output_ids[0][len(input_ids[0]) :])
responses.append(response)
debug_info["sample_responses"] = list(
zip(sample_prompts[:5], responses)
)
full_texts = [p + r for p, r in zip(sample_prompts[:5], responses)]
tokens = tokenizer(full_texts, padding=True, return_tensors="pt").to(
device
)
rewards = reward_model(tokens["input_ids"], tokens["attention_mask"])
debug_info["rewards"] = rewards.cpu().tolist()
debug_info["response_lengths"] = [len(r) for r in responses]
debug_info["unique_words"] = [len(set(r.split())) for r in responses]
return debug_info
