Instruction Following Evaluation › Worked Example: Building an Evaluation Pipeline
Language AI HandbookAll chapters Instruction Following Evaluation › Worked Example: Building an Evaluation PipelineOne section · Sections in this chapter5 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To make the abstract concepts concrete, let's walk through a complete evaluation pipeline for a hypothetical instruction-tuned model. Suppose we are comparing two models, Model A (a 7B parameter model) and Model B (a 13B parameter model), and we want to know which performs better on general-purpose instruction following.
We begin by selecting a benchmark suite. We choose Alpaca Eval for instruction diversity, MT-Bench for multi-turn capabilities, and IFEval for constraint compliance. This three-benchmark approach answers different questions: Alpaca Eval tells us which model users prefer overall, MT-Bench tells us which model handles multi-turn conversations better, and IFEval tells us which model follows explicit instructions more reliably. Using all three ensures we do not miss important capability differences that any single benchmark would obscure.
For Alpaca Eval, we collect 805 responses from each model and run them through GPT-4 as judge with position swapping. Suppose Model A wins 420 comparisons, Model B wins 310, and 75 are ties. The win rate for Model A (excluding ties) is:
That is, Model A wins approximately 57.5% of non-tie comparisons. For MT-Bench, GPT-4 scores responses on a 1-10 scale per turn. Suppose Model A averages 7.2 and Model B averages 6.8 across all 80 multi-turn conversations. For IFEval, automated constraint checking finds Model A satisfies 81% of constraints while Model B satisfies 74%.
import numpy as np
# Worked example: inter-annotator agreement check on 100 Alpaca Eval comparisons
np.random.seed(7)
n_comparisons = 100
rater1_worked = []
rater2_worked = []
for i in range(n_comparisons):
true_quality = np.random.uniform(0, 1)
if true_quality < 0.40:
# Clearly Model A is better; raters agree
rater1_worked.append(1)
rater2_worked.append(1)
elif true_quality < 0.65:
# Model B is better; raters agree
rater1_worked.append(2)
rater2_worked.append(2)
elif true_quality < 0.80:
# Close call; raters may disagree
rater1_worked.append(np.random.choice([1, 2, 3]))
rater2_worked.append(np.random.choice([1, 2, 3]))
else:
# Genuine tie; raters usually agree on tie but not always
rater1_worked.append(3)
rater2_worked.append(np.random.choice([1, 2, 3]))
def cohens_kappa_worked(rater1, rater2):
r1 = np.array(rater1)
r2 = np.array(rater2)
observed_agreement = np.mean(r1 == r2)
categories = np.unique(np.concatenate([r1, r2]))
expected_agreement = sum(
np.mean(r1 == cat) * np.mean(r2 == cat) for cat in categories
)
if expected_agreement == 1:
return 1.0
return (observed_agreement - expected_agreement) / (1 - expected_agreement)
kappa_worked = cohens_kappa_worked(rater1_worked, rater2_worked)
raw_agreement_worked = np.mean(
np.array(rater1_worked) == np.array(rater2_worked)
)Worked Example: Inter-Annotator Agreement Raw agreement: 73.0% Cohen's Kappa: 0.571 Interpretation: Moderate agreement -- results are directionally trustworthy Summary of three-benchmark comparison: Alpaca Eval win rate (Model A): 57.5% MT-Bench score gap: +0.4 for Model A (7.2 vs 6.8) IFEval compliance: 81% vs 74% for Model A Conclusion: Model A outperforms across all three dimensions
The combined picture points clearly toward Model A: it outperforms across all three dimensions. But the margins tell us something more fine-grained. The 57.5% win rate on Alpaca Eval is a real but not decisive advantage on open-ended quality. The 0.4-point gap on MT-Bench is within the typical variance of GPT-4 scoring and should not be over-interpreted without wider confidence intervals. The 7-point constraint compliance gap on IFEval is the most decisive finding, suggesting that Model B struggles with following explicit instructions even when it generates content that users prefer qualitatively.
This pattern tells us something actionable about deployment decisions. If the use case involves structured requests with clear requirements (document formatting, structured data extraction, form completion), Model A is clearly the better choice because it reliably satisfies stated constraints. If the use case is open-ended creative assistance where constraint compliance matters less, the decision is closer, and we should collect additional human evaluation data targeted at that specific use case before committing.
A Kappa around 0.5 to 0.6 in this scenario is typical for open-ended instruction-following evaluation. Close-call comparisons often elicit disagreement because reasonable people weigh quality dimensions differently. The moderate agreement tells us that our evaluation is capturing something real, but that individual comparisons should not be over-interpreted. The model-level win rate (aggregating 805 comparisons) is much more reliable than any individual comparison.