Human Preference Data › Converting to Training Format
Language AI HandbookAll chapters Human Preference Data › Converting to Training FormatOne section · Sections in this chapter8 / 13 ↓
Preference data must be converted to a format suitable for training reward models before it can be used. The standard format pairs each prompt with its chosen and rejected responses, creating a direct learning signal: the reward model should assign higher scores to chosen than to rejected.
This conversion step also provides an opportunity for final quality filtering. You can exclude ties (which provide no directional signal), remove annotations with suspicious timing, weight annotations by annotator reliability if you have quality estimates, or deduplicate prompts to avoid over-representing certain topics in the training data.
def convert_to_training_format(
dataset: PreferenceDataset, exclude_ties: bool = True
) -> list[dict]:
"""
Convert preference samples to the standard training format.
Returns list of dicts with keys: prompt, chosen, rejected
"""
training_data = []
for sample in dataset.samples:
if exclude_ties and sample.preference == "tie":
continue
if sample.preference == "a":
chosen = sample.response_a
rejected = sample.response_b
else:
chosen = sample.response_b
rejected = sample.response_a
training_data.append(
{
"prompt": sample.prompt,
"chosen": chosen,
"rejected": rejected,
"annotator_id": sample.annotator_id,
}
)
return training_data
training_data = convert_to_training_format(dataset)Training samples (excluding ties): 25 Sample training instance: Prompt: Explain quantum computing to a beginner.... Chosen: Quantum computing uses qubits instead of regular bits. While normal bits are eit... Rejected: Quantum computing is a revolutionary paradigm in computational theory that lever...
This format directly supports the reward modeling objective we'll cover in upcoming chapters: the reward model learns to assign higher scalar scores to chosen responses than rejected ones for the same prompt. The training loss penalizes the model whenever its score for the chosen response is lower than its score for the rejected response, driving the model to internalize the quality distinctions captured by the preference data.
The annotator_id field in the training format is valuable for advanced training strategies. You can filter training data to include only high-agreement annotations, weight samples by annotator reliability, or train separate reward models for different annotator subgroups and then ensemble them. Preserving annotator identity in the training format keeps these options open.