Instruction Following Evaluation › Instruction Difficulty
Language AI HandbookAll chapters Instruction Following Evaluation › Instruction DifficultyOne section · Sections in this chapter6 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Not all instructions are equally challenging. Understanding what makes instructions difficult helps us build better training sets and evaluate models more thoroughly. A complete evaluation should include instructions spanning the full range of difficulty to identify where models excel and where they struggle. This understanding also informs curriculum design during training: exposing models to appropriately challenging examples at the right stage of training improves learning efficiency.
Think of instruction difficulty as analogous to the difficulty of a math exam problem. Some problems test single, well-defined concepts; others require synthesizing multiple concepts while managing several constraints simultaneously. A student might excel at single-concept problems while struggling with synthesis problems, and vice versa. Similarly, a model might handle factual recall instructions expertly while failing at multi-constraint creative writing tasks, even though both look like "instruction following" at the surface level.
Understanding difficulty is practically important because it shapes evaluation design. An evaluation set composed entirely of easy instructions will not distinguish between a mediocre model and an excellent one; they will both score near the top. An evaluation set with appropriate difficulty spread reveals performance gaps that matter in deployment. The same logic applies to training data: including only easy instructions produces a model that handles easy cases well but fails when users push harder.
Dimensions of Difficulty
Instruction difficulty emerges from several largely independent dimensions. An instruction can be easy along one dimension while being extremely challenging along another. A simple factual question might require specialized domain knowledge, while a complex multi-step task might involve only common knowledge. Understanding these dimensions helps construct balanced evaluation sets that probe different aspects of model capability:
difficulty_dimensions = {
"knowledge_required": {
"description": "Domain expertise needed to answer correctly",
"examples": {
"easy": "What color is the sky?",
"medium": "Explain the difference between TCP and UDP.",
"hard": "Derive the Euler-Lagrange equation from Hamilton's principle.",
},
},
"reasoning_depth": {
"description": "Number of logical steps required",
"examples": {
"easy": "Is 15 greater than 12?",
"medium": "If all A are B, and all B are C, are all A also C?",
"hard": "Given these 5 clues, determine who owns the fish.",
},
},
"constraint_complexity": {
"description": "Number of constraints the response must satisfy",
"examples": {
"easy": "Write a sentence about dogs.",
"medium": "Write a sentence about dogs using exactly 10 words.",
"hard": "Write a haiku about dogs where each line starts with D.",
},
},
"ambiguity": {
"description": "How underspecified is the instruction",
"examples": {
"easy": "Calculate 25 times 4.",
"medium": "Explain recursion.",
"hard": "Help me with my project.",
},
},
"context_length": {
"description": "Amount of input context to process",
"examples": {
"easy": "Summarize this tweet.",
"medium": "Summarize this article (500 words).",
"hard": "Summarize this legal document (50 pages).",
},
},
}Dimensions of Instruction Difficulty --- KNOWLEDGE REQUIRED --- Domain expertise needed to answer correctly Easy: "What color is the sky?" Medium: "Explain the difference between TCP and UDP." Hard: "Derive the Euler-Lagrange equation from Hamilton's principle." --- REASONING DEPTH --- Number of logical steps required Easy: "Is 15 greater than 12?" Medium: "If all A are B, and all B are C, are all A also C?" Hard: "Given these 5 clues, determine who owns the fish." --- CONSTRAINT COMPLEXITY --- Number of constraints the response must satisfy Easy: "Write a sentence about dogs." Medium: "Write a sentence about dogs using exactly 10 words." Hard: "Write a haiku about dogs where each line starts with D." --- AMBIGUITY --- How underspecified is the instruction Easy: "Calculate 25 times 4." Medium: "Explain recursion." Hard: "Help me with my project." --- CONTEXT LENGTH --- Amount of input context to process Easy: "Summarize this tweet." Medium: "Summarize this article (500 words)." Hard: "Summarize this legal document (50 pages)."
A complete evaluation should sample across all difficulty dimensions, not just one. A model might handle high-knowledge-requirement questions well because it memorized relevant facts during pretraining, but fail on multi-step reasoning despite the individual steps being simple. Conversely, a model with strong reasoning capabilities might produce excellent responses to complex logical puzzles while making basic factual errors on domain-specific questions. These capability profiles are common in practice, because different capabilities emerge from different phases of training and different training data compositions.
The IFEval Benchmark
IFEval (Instruction Following Evaluation) specifically measures whether models follow explicit constraints. Unlike open-ended benchmarks, IFEval instructions contain verifiable requirements. This design philosophy shows an important insight: while overall response quality is subjective and difficult to measure, constraint compliance is objective and automatically verifiable. A response either contains exactly 100 words or it does not; it either includes the required keyword or it does not. This objectivity lets large-scale automatic evaluation without the biases inherent in LLM-as-judge approaches:
# IFEval constraint types
ifeval_constraints = {
"length_constraints": [
"Write a response with exactly 100 words.",
"Your response should be at least 3 paragraphs.",
"Answer in no more than 2 sentences.",
],
"format_constraints": [
"Respond entirely in lowercase.",
"Use bullet points for your answer.",
"Write your response in JSON format.",
],
"keyword_constraints": [
"Include the word 'therefore' in your response.",
"Do not use the word 'the'.",
"Mention 'artificial intelligence' at least twice.",
],
"structural_constraints": [
"Start your response with 'Certainly!'",
"End your response with a question.",
"Include exactly 3 numbered items.",
],
}
# Example IFEval instruction with multiple constraints
complex_instruction = """Write a product description for a smartphone.
Your response must:
1. Be exactly 50-75 words
2. Include the word "innovative" at least once
3. Use no more than 2 sentences
4. End with an exclamation mark"""IFEval Constraint Categories: LENGTH CONSTRAINTS: - Write a response with exactly 100 words. - Your response should be at least 3 paragraphs. - Answer in no more than 2 sentences. FORMAT CONSTRAINTS: - Respond entirely in lowercase. - Use bullet points for your answer. - Write your response in JSON format. KEYWORD CONSTRAINTS: - Include the word 'therefore' in your response. - Do not use the word 'the'. - Mention 'artificial intelligence' at least twice. STRUCTURAL CONSTRAINTS: - Start your response with 'Certainly!' - End your response with a question. - Include exactly 3 numbered items. ------------------------------------------------------------ Example Multi-Constraint Instruction: Write a product description for a smartphone. Your response must: 1. Be exactly 50-75 words 2. Include the word "innovative" at least once 3. Use no more than 2 sentences 4. End with an exclamation mark
IFEval's constraints are automatically verifiable, letting fully automatic evaluation. The verification process requires no subjective judgment: we simply check whether each constraint is satisfied according to its precise definition:
def verify_ifeval_constraints(response, constraints):
"""
Verify whether a response satisfies IFEval-style constraints.
Returns dict of constraint -> (passed, details)
"""
results = {}
for constraint_type, constraint_value in constraints.items():
if constraint_type == "min_words":
word_count = len(response.split())
passed = word_count >= constraint_value
results[f"min_{constraint_value}_words"] = (
passed,
f"Has {word_count} words (need at least {constraint_value})",
)
elif constraint_type == "max_words":
word_count = len(response.split())
passed = word_count <= constraint_value
results[f"max_{constraint_value}_words"] = (
passed,
f"Has {word_count} words (need at most {constraint_value})",
)
elif constraint_type == "must_include":
word = constraint_value.lower()
passed = word in response.lower()
results[f"must_include_{word}"] = (
passed,
f"'{word}' {'found' if passed else 'not found'}",
)
elif constraint_type == "must_exclude":
word = constraint_value.lower()
passed = word not in response.lower()
results[f"must_exclude_{word}"] = (
passed,
f"'{word}' {'not found' if passed else 'found'}",
)
elif constraint_type == "ends_with":
passed = response.strip().endswith(constraint_value)
results[f"ends_with_{constraint_value}"] = (
passed,
f"Ends with '{response.strip()[-5:]}'",
)
return results
# Test a response against constraints
test_response = """The new XPhone Pro is innovative smartphone technology
at its finest. With groundbreaking camera capabilities and all-day battery life,
this device transforms how you capture and share life's moments!"""
constraints = {
"min_words": 20,
"max_words": 50,
"must_include": "innovative",
"ends_with": "!",
}
verification_results = verify_ifeval_constraints(test_response, constraints)
overall_passed = all(passed for passed, _ in verification_results.values())
overall_status_text = (
"All constraints satisfied" if overall_passed else "Some constraints failed"
)Response to verify:
"The new XPhone Pro is innovative smartphone technology
at its finest. With groundbreaking camera capabilities and all-day battery life,
this device transforms how you capture and share life's moments!"
Constraint Verification:
--------------------------------------------------
min_20_words: PASS
Has 29 words (need at least 20)
max_50_words: PASS
Has 29 words (need at most 50)
must_include_innovative: PASS
'innovative' found
ends_with_!: PASS
Ends with 'ents!'
--------------------------------------------------
Overall: All constraints satisfied
IFEval lets measuring instruction-following ability independent of response quality. A model might generate excellent prose but fail basic formatting requirements, revealing a gap in instruction compliance. This separation is useful diagnostically: it distinguishes between models that understand what to do but generate poor content versus models that generate good content but fail to follow explicit directions. Both failure modes exist in practice, and they require different interventions to address.
Difficulty Scoring
We can estimate instruction difficulty before evaluation by analyzing instruction characteristics. This heuristic approach lets automatic categorization of instructions, which is useful for so evaluation sets include appropriate coverage across difficulty levels. While no heuristic can perfectly predict how challenging an instruction will be for a given model, analyzing structural features gives a reasonable approximation that is far better than manual difficulty annotation at scale:
import re
def estimate_instruction_difficulty(instruction):
"""
Heuristic difficulty scoring based on instruction features.
Returns score 1-10 and contributing factors.
"""
factors = {}
# Every instruction has at least a small interpretation burden.
score = 1
instruction_lower = instruction.lower()
# Word count (longer instructions often more complex)
word_count = len(instruction.split())
if word_count > 50:
factors["length"] = "very long instruction"
score += 2
elif word_count > 15:
factors["length"] = "long instruction"
score += 1
# Count explicit constraints
constraint_patterns = [
r"\b\d+\s*-\s*word\b",
r"exactly \d+",
r"at least \d+",
r"no more than \d+",
r"must include",
r"must not",
r"do not use",
r"without using",
r"without changing",
]
constraint_count = sum(
len(re.findall(pattern, instruction_lower))
for pattern in constraint_patterns
)
if constraint_count >= 3:
factors["constraints"] = f"{constraint_count} explicit constraints"
score += 3
elif constraint_count >= 1:
factors["constraints"] = f"{constraint_count} explicit constraint(s)"
score += constraint_count
# Multi-step indicators
step_patterns = [r"first.*then", r"step \d", r"\d\.", r"after that"]
action_words = ["identify", "fix", "optimize", "compare", "summarize"]
action_count = sum(word in instruction_lower for word in action_words)
if action_count >= 2:
factors["multi_step"] = "requires multiple steps"
score += min(action_count, 3)
elif any(re.search(p, instruction_lower) for p in step_patterns):
factors["multi_step"] = "requires multiple steps"
score += 2
if "explain" in instruction_lower:
factors["explanation"] = "requires a clear explanation"
score += 1
# Reasoning indicators
reasoning_words = [
"analyze",
"compare",
"contrast",
"evaluate",
"synthesize",
"critique",
"derive",
"prove",
]
found_reasoning = [w for w in reasoning_words if w in instruction_lower]
if found_reasoning:
factors["reasoning"] = (
f"reasoning required ({', '.join(found_reasoning)})"
)
score += 2
# Domain-specific terms (simple heuristic)
technical_patterns = [
r"\b(algorithm|equation|theorem|hypothesis|coefficient)\b",
r"\b(quantum|molecular|neural|genetic|statistical)\b",
r"\b(code|bugs?|optimi[sz]e|performance)\b",
]
if any(re.search(p, instruction_lower) for p in technical_patterns):
factors["technical"] = "technical domain knowledge required"
score += 2
return min(score, 10), factors
# Test on various instructions
test_instructions = [
"What is 2 + 2?",
"Explain the concept of machine learning to a beginner.",
"Write a 500-word essay analyzing the economic impacts of climate change, including at least 3 specific examples and a counterargument.",
"Given the following code, identify all bugs, fix them, then optimize for performance without changing the output.",
]
difficulty_results = []
for instruction in test_instructions:
score, factors = estimate_instruction_difficulty(instruction)
difficulty_results.append(
{"instruction": instruction, "score": score, "factors": factors}
)Instruction Difficulty Analysis
INSTRUCTION: "What is 2 + 2?"
Difficulty Score: 1/10
No difficulty factors detected (simple instruction)
INSTRUCTION: "Explain the concept of machine learning to a beginner."
Difficulty Score: 2/10
Contributing factors:
- requires a clear explanation
INSTRUCTION: "Write a 500-word essay analyzing the economic impacts of climate chang..."
Difficulty Score: 4/10
Contributing factors:
- long instruction
- 2 explicit constraint(s)
INSTRUCTION: "Given the following code, identify all bugs, fix them, then optimize f..."
Difficulty Score: 8/10
Contributing factors:
- long instruction
- 1 explicit constraint(s)
- requires multiple steps
- technical domain knowledge required
While heuristic, difficulty estimation helps balance evaluation sets and identify where models struggle. By tracking performance across difficulty levels, we can characterize model capabilities more precisely. A model that excels on easy instructions but fails on difficult ones has a different performance profile from a model with consistent performance across difficulty levels, even if their average scores are similar. The former may be a well-tuned small model; the latter is more likely to have broad capability.