Alignment, Preferences, and Oversight

Covers alignment definition, helpfulness vs harmlessness, alignment challenges, alignment approaches overview.

Covers preference collection UI, comparison design, annotator guidelines, preference data quality.

Covers pairwise comparison model, preference probability, Bradley-Terry likelihood, preference strength.

Covers reward model architecture, preference loss function, reward model training, reward model evaluation.

Covers reward hacking examples, distribution shift, over-optimization, reward hacking mitigation.

Covers policy definition, REINFORCE algorithm, policy gradient derivation, variance reduction.

Covers clipped objective, PPO derivation, trust region intuition, PPO implementation.

Covers LLM as policy, action space (tokens), reward assignment, KL penalty importance.

Covers SFT stage, reward model training, PPO fine-tuning, RLHF hyperparameters, RLHF debugging.

Covers KL penalty motivation, KL coefficient selection, adaptive KL, KL effects on training.

Covers DPO motivation, removing reward model, DPO intuition, DPO benefits.

Covers DPO from RLHF objective, optimal policy derivation, DPO loss function, DPO as classification.

Covers DPO data format, DPO loss computation, DPO training procedure, DPO hyperparameters.

Covers IPO formulation, KTO for unpaired feedback, ORPO, cDPO, comparing alignment methods.

Covers AI as annotator, constitutional AI principles, AI preference generation, RLAIF scalability.

Covers iterative DPO, online preference learning, self-improvement loops, alignment stability.

Written principles, critique and revision, AI feedback, evaluation, and failure modes.

Decomposition, debate, recursive supervision, weak-to-strong generalization, and oversight bottlenecks.

Supervising intermediate behavior, specifying policies, resolving conflicts, and testing adherence.

Rater disagreement, cultural pluralism, annotator effects, preference aggregation, and auditability.

Sycophancy, reward tampering, jailbreak pressure, deployment drift, and adversarial evaluation.