Instruction Following Evaluation › Benchmarks for Instruction Following
Language AI HandbookAll chapters Instruction Following Evaluation › Benchmarks for Instruction FollowingOne section · Sections in this chapter2 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Evaluating instruction-tuned models requires benchmarks that capture the breadth of tasks you care about. The challenge is creating evaluation frameworks that reflect real-world usage and produce actionable measurements. Early evaluation relied heavily on existing NLP benchmarks, but the field has developed specialized benchmarks that better reflect real-world instruction following. Anyone developing or deploying instruction-tuned models needs to understand the available benchmarks and the strengths and limitations of each.
Benchmarks serve multiple roles in the development process. During training, they give signal for early stopping and hyperparameter selection. During model comparison, they give a shared vocabulary that lets researchers communicate about model quality without releasing expensive human evaluations. During deployment decisions, they give evidence that a model is appropriate for a given task. But benchmarks are never neutral: the choice of which benchmarks to use shapes the direction of research, because teams optimize for metrics they can measure. A benchmark that misses important capabilities will lead to models that neglect those capabilities.
The field broadly categorizes evaluation benchmarks into three tiers: standard NLP benchmarks that measure underlying capabilities, instruction-specific benchmarks that measure response quality on diverse tasks, and constraint-verification benchmarks that measure compliance with explicit requirements. A complete evaluation strategy samples from all three tiers, using each to answer a different question about model behavior.
Standard NLP Benchmarks
Before instruction-specific benchmarks emerged, we evaluated instruction-tuned models on established benchmarks. While these do not directly measure instruction following, they give useful baselines for comparing model capabilities. The reasoning behind using these traditional benchmarks is straightforward: if a model cannot show strong performance on well-defined tasks with clear correct answers, we have little reason to expect it will handle the more ambiguous challenge of open-ended instruction following. Competence on structured tasks is a necessary, though not sufficient, condition for effective instruction following.
MMLU (Massive Multitask Language Understanding) tests knowledge across 57 subjects from elementary mathematics to professional law. Questions are multiple-choice, making evaluation straightforward. The breadth of MMLU's subject coverage makes it particularly useful for assessing whether a model has acquired broad factual knowledge during training, which forms a foundation for responding helpfully to diverse requests from users:
# Example MMLU question structure
mmlu_example = {
"question": "What is the capital of Australia?",
"choices": ["Sydney", "Melbourne", "Canberra", "Perth"],
"answer": "C",
"subject": "geography",
}
# For instruction-tuned models, we format as a prompt
prompt = f"""Answer the following multiple choice question.
Question: {mmlu_example["question"]}
A) {mmlu_example["choices"][0]}
B) {mmlu_example["choices"][1]}
C) {mmlu_example["choices"][2]}
D) {mmlu_example["choices"][3]}
Answer:"""Answer the following multiple choice question. Question: What is the capital of Australia? A) Sydney B) Melbourne C) Canberra D) Perth Answer:
This formatting is necessary for evaluation. By structuring the task as a prompt, we can parse the model's next token (e.g., "C") to determine accuracy. Evaluating instruction-tuned models requires translating structured tasks into natural language prompts that match how users interact with the model. This translation step itself introduces variability: different prompt phrasings can yield different accuracy scores. This shows the sensitivity of instruction-tuned models to precise wording. Research has shown that MMLU accuracy can vary by several percentage points based solely on prompt phrasing, which complicates direct comparisons across papers that use different prompt templates.
HellaSwag tests commonsense reasoning through sentence completion, while TruthfulQA evaluates whether models generate truthful answers rather than plausible-sounding falsehoods. These benchmarks tell us about model capabilities but do not directly measure whether a model follows arbitrary instructions. The distinction matters because a model might possess extensive knowledge (scoring well on MMLU) and strong reasoning abilities (performing well on HellaSwag) yet still struggle to understand what users want when they phrase requests in natural, conversational language. Knowledge and instruction-following are partially separable skills: knowledge is learned during pretraining, while instruction-following is shaped primarily by fine-tuning.
Instruction-Specific Benchmarks
The field has developed benchmarks specifically designed to evaluate instruction following. These focus on the quality of responses to diverse, open-ended instructions. We develop specialized benchmarks because instruction following requires more than knowledge retrieval or logical reasoning. It requires understanding the user's intent, adapting tone and format appropriately, and maintaining coherence across complex requests. A model that excels at standard NLP tasks may still produce responses that are technically accurate but fail to address what the user wanted.
Alpaca Eval contains 805 instructions covering a range of tasks. The key innovation is using an automatic evaluator (typically GPT-4) to compare model outputs against a reference model's outputs. This approach addresses the scalability problem inherent in human evaluation while still creating preference-based rankings that correlate reasonably well with human judgments:
# Sample of Alpaca Eval instruction categories
alpaca_categories = {
"brainstorming": "Give me 5 creative names for a coffee shop",
"classification": "Classify the sentiment of this review: 'The food was okay but the service was terrible'",
"code": "Write a Python function to find the nth Fibonacci number",
"creative_writing": "Write a short story about a robot learning to paint",
"extraction": "Extract all dates mentioned in the following text",
"general_qa": "What causes the northern lights?",
"math": "If a train travels 120 km in 2 hours, what is its average speed?",
"rewriting": "Rewrite this sentence in passive voice: 'The cat chased the mouse'",
"summarization": "Summarize the main points of this article",
}Alpaca Eval covers diverse instruction types: BRAINSTORMING Example: Give me 5 creative names for a coffee shop CLASSIFICATION Example: Classify the sentiment of this review: 'The food was okay but the service was terrible' CODE Example: Write a Python function to find the nth Fibonacci number CREATIVE_WRITING Example: Write a short story about a robot learning to paint EXTRACTION Example: Extract all dates mentioned in the following text GENERAL_QA Example: What causes the northern lights? MATH Example: If a train travels 120 km in 2 hours, what is its average speed? REWRITING Example: Rewrite this sentence in passive voice: 'The cat chased the mouse' SUMMARIZATION Example: Summarize the main points of this article
These categories illustrate the move from narrow tasks to broad capabilities. The model must handle creative writing as well as logic and information extraction within a single interface. What makes this benchmark particularly useful is its recognition that users do not restrict themselves to a single task type. A model deployed as a general assistant must move between creative content and analytical work, then answer factual questions when needed, often within a single conversation session.
MT-Bench (Multi-Turn Benchmark) evaluates models on 80 multi-turn conversations across 8 categories: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. The multi-turn aspect is important because real conversations require maintaining context. This benchmark addresses a necessary gap in single-turn evaluations: the ability to build coherently on previous exchanges, remember earlier constraints, and integrate new information without contradicting prior responses:
# MT-Bench multi-turn example
mt_bench_example = {
"category": "math",
"turns": [
{
"turn": 1,
"user": "What is the sum of all prime numbers less than 20?",
"expected_capabilities": ["identify primes", "perform addition"],
},
{
"turn": 2,
"user": "Now exclude any primes that are also Fibonacci numbers.",
"expected_capabilities": [
"identify Fibonacci numbers",
"set subtraction",
"recall previous answer",
],
},
],
}MT-Bench Multi-Turn Example: Category: math Turn 1: What is the sum of all prime numbers less than 20? Requires: identify primes, perform addition Turn 2: Now exclude any primes that are also Fibonacci numbers. Requires: identify Fibonacci numbers, set subtraction, recall previous answer
The second turn tests whether the model can build on its previous response while integrating new constraints. This design reflects how humans use conversational AI systems: they start with an initial request, then refine or extend it based on the model's response. A model that excels at isolated single-turn responses but fails to maintain coherence across multiple turns will frustrate users who expect continuous understanding across the conversation.
LMSYS Chatbot Arena takes a different approach: users compare model outputs head-to-head without knowing which model produced which response. This crowdsourced evaluation gives authentic preference data but is slow and expensive to collect at scale. The strength of this approach is that it shows real-world usage patterns. Rather than relying on predetermined instructions that may not reflect actual user needs, Arena captures real queries and preferences in a natural setting. The Elo rating system used to aggregate Arena results handles the fact that not all model pairs are compared equally, creating a global ranking from incomplete pairwise data.
Benchmark Limitations
No benchmark perfectly captures instruction-following ability. Standard benchmarks like MMLU test knowledge but not the ability to follow arbitrary formatting requests. Instruction benchmarks like Alpaca Eval depend heavily on the automatic evaluator's biases, which we will examine in detail in the section on LLM-as-judge. Arena-style benchmarks reflect user preferences but may favor verbosity or stylistic flourishes over correctness.
There is also the problem of benchmark contamination. As models are trained on increasingly large datasets scraped from the web, the probability that evaluation examples appear in training data increases. A model that has memorized MMLU answers during pretraining will score higher on MMLU without having stronger reasoning abilities. The research community has responded by developing held-out benchmarks and using contamination detection tools, but this remains an arms race between increasingly large training datasets and evaluation integrity.
A reliable evaluation strategy uses multiple benchmarks, combining automatic metrics for rapid iteration with periodic human evaluation for ground truth. No single number summarizes model quality; you need a dashboard of metrics that each illuminate a different aspect of behavior.