DPO Implementation › DPO Data Format
Language AI HandbookAll chapters DPO Implementation › DPO Data FormatOne section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Every machine learning algorithm begins with data, and DPO is no exception. The specific structure of preference data directly encodes the learning signal that DPO exploits. Understanding the data format deeply is the first step toward understanding why the algorithm works, and it prevents a category of subtle bugs that arise when training data is improperly prepared.
DPO training requires preference data organized as triplets: a prompt paired with two completions where one is preferred over the other. This structure directly reflects the pairwise comparison paradigm we discussed in the Human Preference Data chapter. The basic insight is that humans are often better at comparing two options than they are at rating a single option on an absolute scale. By framing alignment as a comparison task, preference data captures reliable signal even when absolute quality is hard to define.
Think of the preference triplet as a trial in a competition. The judge (the human annotator) watches two contestants (the two completions) perform the same task (respond to a given prompt) and declares a winner. The training objective then asks the model to internalize what made the winner better than the loser. Critically, the model never receives an explicit description of the judgment criteria: it infers the criteria from many such comparisons, just as a student learns a teacher's grading standards by seeing many graded assignments rather than by reading a rubric.
The quality gap between chosen and rejected responses need not be dramatic. Some of the most valuable preference pairs differ in subtle ways: one response is slightly more concise, slightly more accurate, or slightly more helpful than the other. These fine-grained comparisons teach the model fine-grained preferences that are difficult to specify explicitly. Gross differences, like a correct answer versus a wrong answer, are also useful but teach the model less about the subtleties of quality that distinguish good models from great ones.
The Preference Triplet Structure
Each training example consists of three components:
- Prompt: The input context or instruction that elicits a response
- Chosen response: The completion preferred by human annotators (the "winner")
- Rejected response: The completion deemed less desirable (the "loser")
Unlike instruction tuning where you only need prompt-response pairs, DPO explicitly requires contrastive examples. The model learns what a good response looks like and what makes it better than alternatives. This contrastive learning signal is basic to how DPO operates: by presenting the model with pairs of responses to the same prompt, we provide direct supervision about the relative quality of different outputs. The model can then internalize these comparisons and generalize the underlying preference criteria to new situations.
The key insight is that the same prompt appearing in both the chosen and rejected sequences is not redundant. It is structurally needed. Because both responses answer the same question, any quality difference between them must come from the response itself, not from variation in the question. This controlled comparison is what allows DPO to isolate and learn the preference signal cleanly.
# uv pip install torch transformers peft datasets matplotlib tqdm scipy numpy
# Standard DPO data format
preference_example = {
"prompt": "Explain quantum entanglement in simple terms.",
"chosen": "Quantum entanglement is like having two magic coins that always land on opposite sides, no matter how far apart they are. When you flip one and get heads, the other instantly becomes tails. Scientists don't fully understand why, but this connection exists even across large distances.",
"rejected": "Quantum entanglement is a phenomenon where particles become correlated. The quantum state of each particle cannot be described independently. Measurements on entangled particles are correlated.",
}The chosen response here is more accessible and uses a helpful analogy, while the rejected response is technically accurate but less engaging for a general audience. This kind of subtle preference is exactly what DPO learns to capture.
Dataset Structure for Training
In practice, you will work with datasets containing thousands of such triplets. A dataset of this size lets the model encounter many different preference dimensions. Comparisons may judge helpfulness and accuracy, response length, tone, safety, or other qualities. Diversity of prompt types and preference criteria is important because DPO learns a single unified policy from all these comparisons simultaneously. A dataset that is too narrow in its coverage will produce a model that is well-aligned in that narrow domain but poorly aligned elsewhere.
The Hugging Face datasets library provides a natural format for managing preference data. Its lazy loading, efficient serialization, and direct integration with the transformers ecosystem make it the practical choice for most DPO training pipelines. For smaller datasets you can use Python lists, but the datasets library becomes increasingly valuable as your dataset grows beyond memory limits.
# Create a minimal preference dataset for demonstration
preference_data = [
{
"prompt": "Write a haiku about programming.",
"chosen": "Silent keystrokes fall\nBugs emerge from tangled code\nCoffee fuels the night",
"rejected": "Programming is fun\nI like to write code all day\nComputers are cool",
},
{
"prompt": "What's the capital of France?",
"chosen": "The capital of France is Paris. It's located in the north-central part of the country along the Seine River and serves as the nation's cultural, economic, and political center.",
"rejected": "Paris.",
},
{
"prompt": "How do I make scrambled eggs?",
"chosen": "Crack 2-3 eggs into a bowl, add a splash of milk, and whisk until combined. Heat butter in a non-stick pan over medium-low heat. Pour in the eggs and gently stir with a spatula as they cook, pushing from the edges toward the center. Remove from heat while still slightly wet. They'll finish cooking from residual heat. Season with salt and pepper.",
"rejected": "Put eggs in a pan and stir them around until they're done. Add salt if you want.",
},
{
"prompt": "Explain why the sky is blue.",
"chosen": "The sky appears blue because of Rayleigh scattering. Sunlight contains all colors, but when it hits Earth's atmosphere, shorter blue wavelengths scatter more than longer red wavelengths. This scattered blue light reaches our eyes from all directions, making the sky look blue.",
"rejected": "The sky is blue because of how light works in the atmosphere.",
},
]from datasets import Dataset
# Convert to Hugging Face Dataset
dataset = Dataset.from_list(preference_data)
dataset_size = len(dataset)
features = list(dataset.features.keys())Dataset size: 4 examples Features: ['prompt', 'chosen', 'rejected']
The dataset contains 4 examples with the expected features: prompt, chosen response, and rejected response. This structure is ready for processing into the format required for the DPO loss. While 4 examples is purely illustrative, the same code structure scales directly to the tens of thousands of examples used in real alignment runs.
Tokenization for DPO
Tokenization is where many DPO implementations go wrong. The challenge is not tokenization itself, which is straightforward, but correctly identifying which tokens in the resulting sequence should contribute to the loss. This distinction matters enormously because computing the loss over the wrong tokens trains the model in the wrong direction.
A important implementation detail is how we tokenize preference data. Unlike standard language model training where we tokenize single sequences, DPO requires processing the prompt-response pairs such that we can compute log probabilities only over the response tokens. Including prompt tokens in the DPO loss calculation would contaminate the signal, as the loss measures how the model's response probabilities differ from the reference model. The prompt is shared between both the chosen and rejected responses, so any probability differences there would be noise rather than meaningful preference information.
Think of the tokenized sequence as a document with two sections marked by a dividing line. Everything to the left of the line is the shared, fixed prompt, which the loss function ignores. Everything to the right is the response: this is what we are training the model to prefer. The response mask is how we encode the position of that dividing line in a form that PyTorch can use to zero out the contributions of prompt tokens when summing up the loss.
Tokenization must accomplish two goals. First, it must concatenate the prompt and response into a single sequence that the autoregressive model can process. Second, it must track exactly where the prompt ends and the response begins, so that we can mask out the prompt tokens when computing the loss. The response mask tracks this boundary.
from transformers import AutoTokenizer
## Load tokenizer
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token # GPT-2 doesn't have a pad token
def tokenize_preference_pair(example, tokenizer, max_length=512):
"""
Tokenize a preference pair for DPO training.
Returns input_ids and a mask showing which tokens are from the response.
"""
prompt = example["prompt"]
chosen = example["chosen"]
rejected = example["rejected"]
# Tokenize prompt to find where response starts
prompt_tokens = tokenizer(prompt, add_special_tokens=False)
prompt_length = len(prompt_tokens["input_ids"])
# Tokenize full sequences (prompt + response)
chosen_full = tokenizer(
prompt + chosen,
max_length=max_length,
truncation=True,
padding="max_length",
return_tensors="pt",
)
rejected_full = tokenizer(
prompt + rejected,
max_length=max_length,
truncation=True,
padding="max_length",
return_tensors="pt",
)
# Create masks: 1 for response tokens, 0 for prompt and padding
chosen_mask = chosen_full["attention_mask"].clone()
chosen_mask[0, :prompt_length] = 0 # Mask out prompt tokens
rejected_mask = rejected_full["attention_mask"].clone()
rejected_mask[0, :prompt_length] = 0
return {
"chosen_input_ids": chosen_full["input_ids"].squeeze(0),
"chosen_attention_mask": chosen_full["attention_mask"].squeeze(0),
"chosen_response_mask": chosen_mask.squeeze(0),
"rejected_input_ids": rejected_full["input_ids"].squeeze(0),
"rejected_attention_mask": rejected_full["attention_mask"].squeeze(0),
"rejected_response_mask": rejected_mask.squeeze(0),
"prompt_length": prompt_length,
}# Tokenize one example
example = preference_data[0]
tokenized = tokenize_preference_pair(example, tokenizer)
prompt_len = tokenized["prompt_length"]
chosen_seq_len = tokenized["chosen_attention_mask"].sum().item()
chosen_resp_len = tokenized["chosen_response_mask"].sum().item()
rejected_resp_len = tokenized["rejected_response_mask"].sum().item()Prompt length: 7 tokens Chosen sequence length: 27 tokens (non-padding) Chosen response tokens: 20 Rejected response tokens: 17
The response mask is necessary: it tells us which token positions should contribute to the DPO loss. We only want to compare log probabilities over the response tokens, not the shared prompt. Without this mask, the loss would be dominated by the prompt tokens, which are identical for both chosen and rejected responses and therefore carry no useful preference signal. A model trained without proper masking would learn nothing meaningful from the preference data.