Evaluation Fundamentals › A Worked Example: Designing an Evaluation for a Summarization Model

Let's walk through how to apply these principles to design an evaluation for a model intended to summarize news articles.

Step 1: Define the evaluation goals. We want to measure (a) factual accuracy, (b) coverage of key points, (c) conciseness, and (d) fluency. These correspond to different failure modes: a model might be fluent but factually wrong, or accurate but verbose, or concise but missing key information. Each of these dimensions must be measured separately, because they are not correlated and a model that excels on one may fail on another.

Step 2: Choose metrics for each goal. Factual accuracy requires human annotation or an automated fact-checking system; ROUGE scores will not catch factual errors because they measure token overlap rather than semantic truth. Coverage can be approximated by ROUGE recall (does the summary contain the key n-grams from the article?). Conciseness can be measured by compression ratio. Fluency can be measured by a language model perplexity score or by human raters.

Step 3: Construct the test set. We collect 500 news articles from the target deployment domain (not from CNN/DailyMail if we are not deploying on CNN/DailyMail). We verify these articles are not in the model's training data by checking against dates and source identifiers. We collect reference summaries from professional editors.

Step 4: Plan statistical analysis. With 500 examples, we can detect differences of approximately 2 percentage points in ROUGE-1 recall at 95% confidence. We document this power calculation before running the evaluation, committing to the sample size before seeing results.

Step 5: Establish controls. All models are evaluated with the same prompt template, the same decoding temperature, and the same maximum output length. We document these choices explicitly. When comparing multiple models, we randomize evaluation order to avoid any systematic effects.

Step 6: Conduct human evaluation on a sample. We cannot afford human evaluation on all 500 examples, so we randomly sample 50 and have three annotators rate factual accuracy and coverage on a 5-point Likert scale. We compute inter-annotator agreement before proceeding. If agreement is too low (kappa below 0.4), we revise the annotation guidelines and re-annotate.

Step 7: Interpret results in context. A 3-point improvement in ROUGE-1 that does not correspond to improvement in human factual accuracy ratings suggests the model is adding words that match reference summaries without being more accurate. We report both metrics and flag the discrepancy.

This kind of multi-metric, principled evaluation design is what distinguishes useful evaluation from evaluation theater.