Evaluation Prompt Engineering: Designing Reliable LLM Judges › Worked Example: Designing a Factual Accuracy Evaluator
Language AI HandbookAll chapters Evaluation Prompt Engineering: Designing Reliable LLM Judges › Worked Example: Designing a Factual Accuracy EvaluatorOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us walk through the full process of designing an evaluation prompt from scratch for a specific task: evaluating the factual accuracy of responses to knowledge questions. This is one of the most common evaluation tasks in language model development, and it illustrates the full range of design decisions we have covered.
Task description: We are building an evaluation pipeline for a question-answering model. Our responses are answers to factual questions (for example, "What year did the Berlin Wall fall?"). We want to measure whether the model's response is factually correct, without crediting fluent-sounding but incorrect answers.
Step 1: Define the criterion.
We will evaluate a single dimension (factual accuracy) on a 1-4 scale:
- 4: The response answers the question correctly with no factual errors.
- 3: The response answers the question mostly correctly but includes a minor error or imprecision that does not change the core answer.
- 2: The response attempts to answer the question but contains a significant factual error that affects the core answer.
- 1: The response is factually wrong, fabricated, or refuses to answer without basis.
We chose a 1-4 scale rather than 1-5 deliberately: even-numbered scales force the judge away from a neutral midpoint, which is useful for factual accuracy where responses should generally be either correct or incorrect. An odd-numbered scale would allow the judge to score borderline cases as "3 out of 5 (neutral)" even when the response is substantively wrong or right.
Step 2: Write the task description.
You are an expert fact-checker evaluating language model responses to factual questions.
For each response, you will assess factual accuracy on a 1-4 scale:
4: Correct. The response answers the question accurately with no errors.
3: Mostly correct. Minor imprecision present but core answer is right.
2: Partially correct. A significant error affects the core answer.
1: Wrong. The response is factually incorrect, fabricated, or refuses to answer.
You must base your verdict ONLY on factual accuracy, not fluency, length, or tone.
After the response, provide:
- A one-sentence explanation of your verdict
- Your numeric score on a line by itself: SCORE: [1/2/3/4]
The explicit exclusion clause ("ONLY on factual accuracy, not fluency, length, or tone") is important. Without it, judges tend to conflate high-quality writing with factual accuracy, giving higher scores to well-written wrong answers than to awkwardly-written correct ones. The exclusion clause is a direct intervention against the surface-quality confound.
Step 3: Create two few-shot examples.
Example 1 (score 4):
Question: In what year did the Berlin Wall fall?
Response: The Berlin Wall fell in 1989.
Explanation: The response correctly identifies 1989 as the year of the Wall's fall.
SCORE: 4
Example 2 (score 2):
Question: What is the speed of light in a vacuum?
Response: The speed of light is approximately 300,000 miles per second.
Explanation: The speed of light is approximately 300,000 kilometers per second (not miles).
The order of magnitude is correct but the unit is wrong, which constitutes a significant error.
SCORE: 2
The second example is a borderline case: the response is not fabricated, and it gets the order of magnitude right, but the unit error makes it factually wrong in a way that matters. The explicit reasoning explains to the judge why this is a 2 rather than a 1 (which would be reserved for responses that are completely wrong) or a 3 (which would be reserved for minor imprecisions that do not affect the core answer). This kind of explicit reasoning in examples teaches the judge how to handle analogous borderline cases across the full evaluation set.
Step 4: Test sensitivity and consistency.
Before deploying this prompt, we should:
- Test it against 20-30 (question, response, human verdict) examples and compute agreement between the judge and human raters.
- Create three paraphrase variants of the task description and verify that agreement across variants is high (kappa > 0.8).
- Run the judge three times on the same inputs at temperature 0 and verify perfect consistency (100% agreement across runs).
- Examine the score distribution and verify it covers the full 1-4 range, not just 3-4.
This example is simpler than production evaluation prompts, which typically involve longer responses, more fine-grained criteria, and domain-specific knowledge. But it illustrates the core principle: explicit criterion definitions, concrete scale anchors, and empirical testing are not optional features of evaluation prompt design. They are the difference between a judge that measures what you care about and one that measures whatever is easiest to measure.