Instruction Data Creation › Human Annotation

Human annotation remains the gold standard for instruction data quality. When a skilled annotator writes an instruction and its response, they bring world knowledge, common sense, and an intuitive understanding of what makes a response helpful. The earliest instruction-tuned models, including InstructGPT from OpenAI, relied heavily on human-written demonstrations. This reliance on human expertise highlights that instruction following depends on human communication patterns and preferences that automated methods struggle to capture.

Think of human annotation as setting the ceiling for data quality. Every other method in this chapter attempts to approach that ceiling more efficiently, accepting some quality degradation in exchange for scale. Human annotators can handle ambiguous instructions, produce careful multi-step reasoning, catch errors in their own output, and write natural responses. These capabilities are difficult to replicate programmatically. Annotation programs aim to collect data while preserving the tacit knowledge that skilled communicators apply automatically.

The key insight is that annotation quality depends critically on the annotation process itself, not just on the skill of individual annotators. Even highly capable annotators produce inconsistent, low-quality data when given poor guidelines, inadequate feedback, or incentive structures that prioritize speed over quality. The most successful annotation programs invest heavily in guideline development, annotator training, and ongoing quality control, treating the annotation process as a product that requires iteration and improvement.

The challenge of human annotation is ultimately economic. Human attention is expensive, and the volume of instruction data needed to train a capable assistant is large. This tension between quality and scale motivates everything that follows in this chapter.

Creating effective annotation guidelines requires balancing specificity with flexibility. Guidelines that are too rigid produce formulaic responses that lack naturalness. Guidelines that are too loose produce inconsistent data where annotators interpret tasks differently. Finding the right balance is itself a design challenge that requires iteration and feedback from annotators who work with the guidelines in practice. This feedback loop is essential: annotators discover ambiguities in guidelines that guideline writers never anticipated, and incorporating that feedback improves the quality of subsequent annotation.

Effective guidelines typically specify the following dimensions of response quality:

• Response format expectations: Should responses include explanations, or just direct answers? Should they acknowledge uncertainty? These choices shape the personality and communication style of the resulting model.
• Tone and style: Formal or conversational? Verbose or concise? The tone established in training data carries through to the model's behavior with you.
• Handling edge cases: What should annotators do with ambiguous requests, harmful queries, or questions outside their expertise? Clear guidance prevents annotators from improvising inconsistently.
• Quality thresholds: What makes a response "good enough" versus requiring revision? Concrete examples of acceptable and unacceptable responses help calibrate annotator judgment.

The Dolly dataset from Databricks provides a useful example. Its guidelines asked annotators to write creative and brainstorming tasks as well as question-answering tasks. Classification and summarization covered additional instruction types; information extraction did too. Annotators retained freedom within each category. This approach recognized that creativity flourishes within constraints, and that annotators need enough structure to be consistent while retaining enough freedom to produce natural, varied examples. Databricks also encouraged annotators to write about topics they personally found interesting or knew well, producing more authentic and detailed responses than tasks assigned at random would generate.

Good annotation guidelines also specify what to avoid. Instructions that duplicate content from well-known copyrighted sources, instructions designed to elicit harmful responses, and responses that reflect personal opinions as facts all represent pitfalls that explicit guidance can prevent. Annotators are more likely to avoid problematic patterns when those patterns are named and explained than when they are expected to deduce the right approach from general principles.

Human annotation introduces human variability. Different annotators bring different writing styles, knowledge levels, and interpretations of guidelines. Quality control mechanisms help maintain consistency across a diverse pool of contributors and ensure that the resulting dataset reflects the intended quality standards rather than the idiosyncrasies of individual annotators.

The most important quality control mechanism is the feedback loop. Annotators need timely, specific feedback on their work to improve and to understand how the guidelines apply to specific cases. Feedback that arrives days after annotation is completed is less useful than immediate review. Many annotation programs use a tiered model where senior annotators review a sample of each contributor's work. This provides written feedback on specific examples. This feedback is then discussed in group calibration sessions where annotators can ask questions and ensure they share a common interpretation of the guidelines.

The main quality control practices in annotation programs include:

• Inter-annotator agreement: Having multiple annotators label the same examples reveals disagreements that indicate unclear guidelines or difficult cases. When two annotators produce very different responses to the same instruction, it signals either an ambiguous task or a need for guideline refinement.
• Expert review: Senior annotators review a sample of outputs. This provides feedback and identifying systematic issues. This creates a feedback loop where common problems are caught early and addressed through guideline updates or additional training.
• Iterative refinement: Guidelines evolve based on observed problems, with regular calibration sessions to realign annotator understanding. This ongoing process acknowledges that annotation guidelines are living documents that improve through use.
• Rejection and revision cycles: Rather than accepting all submitted work, high-quality annotation programs reject examples that fall below quality thresholds, requiring revision or replacement. This creates accountability and prevents the dataset from gradually accumulating low-quality examples.

Inter-annotator agreement deserves special attention because it serves two purposes simultaneously. Measuring agreement identifies areas where annotators disagree, signaling guideline gaps. But the process of having annotators attempt the same tasks also reveals the inherent difficulty of some instructions: if even expert annotators cannot agree on the best response, that instruction type may not belong in the training set at all.

Human annotation is expensive. Professional annotators cost $15-50+ per hour depending on the task complexity and required expertise. A single high-quality instruction-response pair might take 5-15 minutes to create, putting the cost per example between $1-15. These costs accumulate quickly when building datasets of any meaningful size. When you factor in guideline development, annotator training, quality review overhead, and revision cycles, the true cost per usable example is often two to three times the direct annotation cost.

This cost creates a fundamental tradeoff: you can have either a small dataset of exceptional quality or a larger dataset of variable quality. The InstructGPT paper used only about 13,000 human demonstrations, a tiny dataset by pre-training standards, but each example was carefully crafted by selected contractors who underwent extensive training. This choice reflected a bet that quality would matter more than quantity for teaching instruction-following behavior. That bet turned out to be correct: the model trained on 13,000 high-quality demonstrations substantially outperformed the model trained on pre-training data alone, despite the enormous size difference.