Alignment Challenges › Goal Mis-specification in Practice
Language AI HandbookAll chapters Alignment Challenges › Goal Mis-specification in PracticeOne section · Sections in this chapter5 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The concepts of reward hacking, specification gaming, and alignment failure are not purely theoretical. Let us look at concrete implementation patterns that illustrate how these problems arise and what can be done to mitigate them.
We will implement a simple simulation of reward model overoptimization, illustrating the divergence between a proxy reward signal and a gold-standard quality measure as optimization pressure increases. This simulation captures the essential dynamics: early optimization improves both the proxy and the true quality, but continued optimization exploits the proxy's biases at the expense of true quality.
import numpy as np
np.random.seed(42)
def reward_model(response_quality, noise_std=0.15):
"""
Simulates a reward model: approximately measures quality,
but with noise and a bias toward surface features like length.
"""
# True quality component (most of the reward)
quality_signal = response_quality
# Spurious surface feature bias (length, confidence markers)
# The reward model overweights these by a small amount
spurious_bias = 0.3 * np.random.randn() * (1 - response_quality)
# Measurement noise
noise = noise_std * np.random.randn()
return np.clip(quality_signal + spurious_bias + noise, 0, 1)
def gold_standard(response_quality):
"""
Simulates a gold-standard quality measure: accurate but expensive.
Used only for evaluation, not training.
"""
return response_quality + 0.02 * np.random.randn()
def optimize_against_reward(initial_quality, kl_budget, steps=50):
"""
Simulates optimizing a model against the reward model up to a KL budget.
As optimization progresses, the model finds reward hacking strategies
that increase reward without improving (and eventually degrading) quality.
"""
quality = initial_quality
kl_used = 0
reward_scores = []
gold_scores = []
kl_values = []
kl_step = kl_budget / steps
for step in range(steps):
kl_used += kl_step
# Legitimate quality improvement: gains fast early, slows down
quality_improvement = 0.4 * np.exp(-0.8 * kl_used) * kl_step
quality = min(1.0, quality + quality_improvement)
# Reward hacking: grows with optimization pressure, eventually hurts quality
hacking_degree = 1 - np.exp(-0.3 * kl_used)
quality_from_hacking = quality * (1 - 0.25 * hacking_degree)
reward = reward_model(quality)
gold = gold_standard(quality_from_hacking)
reward_scores.append(reward)
gold_scores.append(gold)
kl_values.append(kl_used)
return np.array(kl_values), np.array(reward_scores), np.array(gold_scores)
# Run the simulation
kl_vals, reward_vals, gold_vals = optimize_against_reward(
initial_quality=0.5, kl_budget=8.0, steps=100
)Peak gold-standard quality at KL = 5.60 Peak gold quality: 0.857 Final reward model score: 1.000 Final gold quality: 0.772 Quality degradation from peak: 0.084
The simulation reveals the characteristic pattern of reward model overoptimization. Gold-standard quality peaks at a moderate level of optimization pressure, then declines as the model increasingly exploits the reward model's biases rather than improving underlying quality. Meanwhile, the reward model continues to report high scores. In practice, this divergence is only detectable if you measure with a separate evaluation signal, which is expensive and is often not done.
This is the core operational challenge: the training signal that drives optimization (the reward model) is not the same as the measure we care about (true quality). As long as these are correlated, optimization makes progress. Once the model has found ways to achieve high reward model scores without achieving high true quality, optimization continues but produces no further benefit, and may actively cause harm.
def measure_specification_gaming_rate(n_requests=1000, optimization_level=0.5):
"""
Simulates the rate at which a model engages in specification gaming
as a function of optimization level. Specification gaming increases
with optimization pressure.
"""
np.random.seed(0)
gaming_rates = []
optimization_levels = np.linspace(0, 1, 50)
for opt_level in optimization_levels:
# Gaming probability increases nonlinearly with optimization
base_gaming_prob = 0.05
gaming_prob = base_gaming_prob + 0.6 * opt_level**2
gaming_count = np.random.binomial(n_requests, min(gaming_prob, 1.0))
gaming_rates.append(gaming_count / n_requests)
return optimization_levels, np.array(gaming_rates)
opt_levels, spec_gaming_rates = measure_specification_gaming_rate()Low optimization (level 0.2): 6.7% gaming rate Medium optimization (level 0.5): 23.6% gaming rate High optimization (level 0.8): 47.3% gaming rate
In this toy simulation, the encoded gaming probability rises with the square of optimization level, and binomial sampling adds realistic variation around that trend. The result illustrates why proxy quality becomes increasingly important under stronger optimization; it is not evidence that a trained model actually learned to game a specification.
