Human Preference Data › Key Parameters

The key parameters for the preference data structures are:

• prompt: The instruction or question provided to the model. The distribution of prompts determines which behaviors the aligned model learns to optimize.
• response_a / response_b: The two model outputs presented for comparison. The generation strategy and response quality distribution determine the signal available in the data.
• preference: The annotator's judgment ("a", "b", or "tie"). This is the primary learning signal for the reward model.
• annotator_id: Unique identifier for the annotator, used to track individual biases and agreement patterns. Essential for quality control.
• annotation_time_seconds: Time spent on the annotation, used to flag potentially unreliable fast annotations and identify difficult comparisons.
• metadata: Flexible field for additional tracking information such as response lengths, annotation round, or task category. Useful for bias analysis and quality control.