DPO Implementation › Key Parameters
Michael BrenndoerferJanuary 2, 20261 min read
Language AI HandbookAll chapters Part 37 of 67Alignment and RLHF
DPO Implementation › Key ParametersOne section · Sections in this chapter9 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for DPO training are:
- beta: Controls the strength of the KL divergence constraint (typically 0.1). Higher values keep the policy closer to the reference model and reduce overfitting risk. Lower values allow faster learning but increase the risk of mode collapse.
- learning_rate: The step size for optimization (typically 5e-7 to 1e-5). DPO requires lower rates than standard supervised fine-tuning because the implicit reward is sensitive to log probability ratio changes.
- batch_size: The number of samples processed per step. Larger batches generally stabilize the loss estimate. An effective batch size of 16-32 preference pairs is a reasonable starting point.
- gradient_accumulation_steps: Number of micro-batches to accumulate before updating weights, used to simulate large batch training under memory constraints.
- num_epochs: Number of passes through the preference dataset (typically 1-3). DPO converges quickly and overfits with too many epochs.
- warmup_ratio: Fraction of training steps used for learning rate warmup (typically 0.05-0.1). Warmup prevents early training instability when the optimizer first encounters large gradients.