DPO Variants › Odds Ratio Preference Optimization (ORPO)

ORPO takes a more radical departure from the DPO framework by eliminating the reference model entirely. Introduced by Hong et al. (2024), ORPO combines supervised fine-tuning with preference optimization into a single training objective, compressing what had been a two-stage pipeline into a single pass.

Think of ORPO as designing a building that gives its own structural support rather than leaning against an adjacent building. DPO leans against the reference model: it measures every change relative to a frozen copy of the initial policy. ORPO's odds formulation gives internal stability by coupling the SFT loss and the odds ratio loss in a way that creates natural constraints on how far the policy can drift.

All previous methods, including RLHF, DPO, IPO, and KTO, require maintaining a reference policy $\pi_{\text{ref}}$. This creates practical complications that matter most for researchers and engineers working with large models under tight resource constraints.

Memory overhead is the most immediate problem. Two copies of the model must reside in GPU memory simultaneously, or the reference model must be frequently loaded and unloaded from disk. For a 70-billion-parameter model, the reference copy alone requires roughly 140 GB of VRAM in BF16 precision, before accounting for gradients, optimizer states, and activations. This pushes many alignment experiments beyond the reach of single-node setups.

Two-stage training adds pipeline complexity. Models typically undergo supervised fine-tuning first to create a sensible reference policy, then preference training relative to that reference. Every bug, hyperparameter choice, and data quality issue must be diagnosed across two separate training runs rather than one.

Computational cost compounds the memory pressure. Every forward pass during preference training requires evaluating both policies: the current trainable policy and the frozen reference. This doubles the compute cost of every gradient step relative to supervised fine-tuning alone.

ORPO asks whether we can eliminate the reference model while still preventing the policy from drifting arbitrarily far from creating sensible outputs. The answer turns out to be yes, provided we design the objective carefully.

Instead of comparing log probabilities to a reference, ORPO uses the odds ratio between chosen and rejected responses within the current policy itself. This shift is significant. Instead of measuring change from a fixed reference, ORPO measures the relative likelihood of responses within the current policy. The odds of generating response $y$ given prompt $x$ are:

$$\text{odds}_\theta(y|x) = \frac{\pi_\theta(y|x)}{1 - \pi_\theta(y|x)}$$

where:

• $\text{odds}_\theta(y|x)$: the odds of generating response $y$ under policy $\theta$
• $\pi_\theta(y|x)$: the probability of the response

The odds representation has a natural interpretation: it tells us how likely the response is compared to everything else. If the odds are 2:1, the response is twice as likely as all alternatives combined. The odds formulation is particularly useful because ratios of odds have clean mathematical properties that are difficult to reach with plain probability ratios.

For a language model, the probability of a specific sequence becomes numerically insignificant as length increases, making direct odds calculation unstable. Consider that even a moderately long sequence might have a probability of $10^{-50}$ or smaller, which would make both the numerator and denominator of the odds calculation problematically small. ORPO instead defines the sequence-level odds as the geometric mean of the token-level odds:

$$\text{odds}_\theta(y|x) \approx \exp\left( \frac{1}{|y|} \sum_{t=1}^{|y|} \log \frac{\pi_\theta(y_t|x, y_{<t})}{1 - \pi_\theta(y_t|x, y_{<t})} \right)$$

where:

• $x$: the input prompt
• $|y|$: the length of the response in tokens
• $y_t$: the token at step $t$
• $y_{<t}$: the sequence of tokens preceding step $t$ (the context)
• $\pi_\theta(y_t|x, y_{<t})$: the probability of the next token given the prompt $x$ and previous tokens $y_{<t}$
• $\exp(\dots)$: the exponentiation to convert average log-odds back to the odds scale

This formulation works at the token level, where probabilities are large enough to be numerically stable, and then aggregates these token-level odds into a sequence-level measure. The averaging by sequence length ensures that longer sequences are not automatically penalized, since we take a geometric mean rather than a product.

The ratio of odds between chosen ($y_w$) and rejected ($y_l$) responses becomes:

$$\text{OR}_\theta(y_w, y_l) = \frac{\text{odds}_\theta(y_w|x)}{\text{odds}_\theta(y_l|x)}$$

where:

• $\text{OR}_\theta$: the odds ratio between the chosen and rejected responses
• $y_w, y_l$: the chosen and rejected responses

This odds ratio captures the relative preference of the current policy for the chosen response over the rejected one. An odds ratio greater than 1 means the policy favors the chosen response; we want to train the policy to increase this ratio.

ORPO combines two components that work together to give both language modeling signal and preference signal. This combination is what allows ORPO to eliminate the separate SFT stage and the reference model.

The first component is the supervised fine-tuning loss on the chosen response:

$$\mathcal{L}_{\text{SFT}} = -\mathbb{E}_{(x, y_w)} \left[ \log \pi_\theta(y_w|x) \right]$$

where:

• $\mathcal{L}_{\text{SFT}}$: the supervised fine-tuning loss component
• $\mathbb{E}_{(x, y_w)}$: the expectation over prompts and chosen responses
• $\pi_\theta(y_w|x)$: the likelihood of the chosen response

This component serves two purposes: it teaches the model to generate fluent, coherent text (the standard language modeling objective), and it gives an anchor that prevents the model from drifting too far from creating sensible outputs. By training on the chosen responses, the model learns what good outputs look like and maintains a probability distribution that assigns reasonable mass to those outputs.

The second component is the odds ratio loss that increases the relative odds of chosen over rejected:

$$\mathcal{L}_{\text{OR}} = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \log \text{OR}_\theta(y_w, y_l) \right) \right]$$

where:

• $\mathcal{L}_{\text{OR}}$: the odds ratio loss component
• $\mathbb{E}_{(x, y_w, y_l)}$: the expectation over the dataset of preference pairs
• $y_w, y_l$: the chosen and rejected responses
• $\log \text{OR}_\theta(y_w, y_l)$: the log odds ratio (log of the ratio of odds)
• $\sigma$: the sigmoid function

This component implements the preference learning objective. By maximizing the log-sigmoid of the log odds ratio, we push the model to make the chosen response relatively more likely than the rejected one. The structure is similar to DPO's loss, but operates on odds ratios within a single policy rather than log-probability ratios between two policies.

The combined ORPO objective is:

$$\mathcal{L}_{\text{ORPO}} = \mathcal{L}_{\text{SFT}} + \lambda \cdot \mathcal{L}_{\text{OR}}$$

where:

• $\mathcal{L}_{\text{ORPO}}$: the total ORPO loss
• $\lambda$: the coefficient weighting the odds ratio loss against the SFT loss

The hyperparameter $\lambda$ controls the trade-off between learning to generate good responses and learning to distinguish good responses from bad ones. Too small a $\lambda$ and the model ignores preferences; too large and the model may sacrifice fluency for preference optimization.

The odds ratio formulation gives implicit regularization without an explicit reference model. To understand why this works, consider what happens during training. The SFT loss pulls the model toward generating the chosen response. The OR loss pushes chosen odds higher relative to rejected odds. Both losses operate on the same policy, creating a coupled optimization.

The key insight is that increasing odds for chosen responses while decreasing odds for rejected responses automatically constrains how much the model can deviate from generating coherent text. If the model tried to maximize the odds ratio by assigning near-zero probability to rejected responses, the SFT loss on chosen responses would suffer because probability mass must be conserved. The model cannot simply declare everything "bad"; it must maintain a coherent probability distribution over all possible responses.

This coupling between the two loss components creates an implicit regularization effect. The SFT loss ensures the model keeps generating reasonable text, while the OR loss ensures it prefers better text to worse text. Neither loss alone would reach both objectives, but together they give a balanced training signal without requiring a separate frozen reference model.

Comparing what DPO and ORPO optimize reveals how they prevent policy drift.

DPO optimizes:

$$\log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$$

where:

• $\pi_\theta, \pi_{\text{ref}}$: the policy and reference model probabilities
• $y_w, y_l$: the chosen and rejected responses

ORPO optimizes:

$$\log \frac{\text{odds}_\theta(y_w|x)}{\text{odds}_\theta(y_l|x)} = \log \text{odds}_\theta(y_w|x) - \log \text{odds}_\theta(y_l|x)$$

where:

• $\log \text{odds}_\theta(y|x)$: the log-odds of a response under the current policy
• $y_w, y_l$: the chosen and rejected responses

DPO measures how much the policy's preference for $y_w$ over $y_l$ has changed relative to the reference. ORPO measures the absolute odds ratio within the current policy. The reference model in DPO is an external anchor. ORPO's SFT component and odds formulation give an alternative form of internal anchoring. The two approaches reach similar regularization effects through fundamentally different mechanisms.