Evaluation Prompt Engineering: Designing Reliable LLM Judges › Evaluation Prompt Design

Designing an evaluation prompt that is reliable and valid requires making deliberate choices about five key components: the task description, the evaluation criteria, the response presentation, the output format, and the persona or role assignment. Each component interacts with the others, and getting them right is an iterative process.

The task description tells the judge what it is doing and why. It should specify the source task (what was the model asked to do?), the dimension being evaluated (what aspect of quality are we measuring?), and the evaluator's role (who is the judge in this scenario?).

A weak task description:

A strong task description:

The stronger version eliminates ambiguity about what is being measured. The word "evaluate" in isolation is underspecified. LLMs are capable of evaluating many properties of text, and without explicit guidance they often weight whatever properties are most salient in their training distribution, which may not match what you care about.

Three elements are consistently worth including in the task description:

• What the model was supposed to do: Include the original user question or task specification. Without this, the judge has no basis for assessing task completion.
• What you are measuring: Be explicit about the dimension (accuracy, fluency, helpfulness, safety, groundedness) and provide a brief definition.
• The standard for judgment: Describe what a high-quality response looks like on that dimension, not just what you are measuring.

The third element is the one most often omitted. "Evaluate factual accuracy" tells the judge what to measure. "A factually accurate response makes only verifiable claims and does not add details not present in the source" tells the judge what the standard is. Without a standard, the judge applies its own internal standard, which is based on its training distribution and may not match yours. The standard clause is especially important for evaluation tasks that are domain-specific or that involve fine-grained judgment calls.

One additional point about task descriptions: they should not describe what a bad response looks like. Focusing the task description on high-quality exemplars helps the judge orient toward positive signal rather than toward catching failures. Prompts that emphasize failure modes tend to sensitize the judge toward that class of errors while underweighting other quality dimensions. This is particularly relevant for safety evaluation, where a prompt that asks the judge to "find unsafe content" will be more sensitive to one type of failure than a prompt that asks it to "assess whether the response is safe for all audiences."

Evaluation criteria are the specific properties the judge should weigh when forming a verdict. Designing good criteria is harder than it looks because criteria must be precise enough to eliminate ambiguity but general enough to apply across the full range of responses the judge will encounter.

Single-dimension vs. multi-dimension evaluation. The most common design mistake is asking a judge to evaluate multiple quality dimensions with a single score. If you ask "Rate this response on accuracy, fluency, and helpfulness" and request a single 1-to-5 score, the judge must aggregate across dimensions using some implicit weighting. That weighting is opaque, unstable, and likely to vary across prompts and judge models. Unless you specifically want an aggregated quality score (and have validated that aggregation reflects your priorities), evaluate dimensions separately and combine them explicitly.

There are legitimate use cases for composite scores. If you are doing a final ranking of models and you have already validated your individual criteria, a weighted composite score is a reasonable summary statistic. But during the design and validation phase, single-dimension evaluation is almost always preferable. It lets you understand what your judge measures, catch criterion drift, and trace unexpected results back to their sources.

Criterion definition. Each criterion should include a name, a brief definition, and examples of high and low performance on that criterion. Vague criterion names like "quality" or "helpfulness" are interpreted differently by different judge instances. Concrete criterion definitions reduce that variance.

Here is an example of a well-specified criterion for factual accuracy:

This rubric-style definition gives the judge concrete anchors at the extremes and a reference point in the middle. It reduces the ambiguity that makes different prompt phrasings produce different results. The explicit description at each scale point forces the judge to map the response onto a specific definition rather than onto its own prior about what a "3 out of 5" means. Evaluators who use unanchored scales frequently observe that different judges interpret the middle of the scale very differently, with some treating it as "acceptable" and others treating it as "mediocre." Anchoring eliminates that ambiguity.

Criterion independence. If your evaluation uses multiple criteria, check whether they are truly measuring different things. Fluency and grammaticality are highly correlated because grammatical text is almost always fluent and fluent text is almost always grammatical. Helpfulness and task completion frequently overlap because completing the assigned task is the primary way a response becomes helpful. Redundant criteria amplify the weight of underlying dimensions without your knowledge: a rubric that lists fluency, readability, grammaticality, and coherence as separate criteria is effectively weighting surface-level language quality four times as heavily as the rubric text implies.

To test for redundancy, run your judge on a sample of 50-100 responses and compute Pearson or Spearman correlation coefficients between the per-criterion scores. Pairs of criteria with correlations above 0.8 are redundant. Decide which label better captures what you care about and drop the other. If you want to preserve both because they occasionally diverge (there are tasks where a response can be fluent but incoherent, for example), retain them but be explicit in your documentation that they are measuring related constructs. Reviewers of your evaluation pipeline will need that context to interpret multi-criterion aggregations correctly.

The heatmap below shows a typical pattern of criterion correlations for a quality assessment rubric. The grouping of surface-level criteria (fluency, grammaticality, readability) in the upper-left corner illustrates why treating them as independent dimensions overstates their individual contributions to overall quality assessment.

How you present the responses to the judge affects verdict quality in ways beyond position bias. The presentation format includes the order of presentation (A before B, or randomized), the labeling of responses ("Response A" vs. "Response 1" vs. "Option Alpha"), the amount of whitespace and structure, and whether you include the original question inline with each response or only at the top of the prompt.

Labeling effects. Research on LLM evaluation has found that certain labels carry implicit quality associations in the training data of judge models. "Response A" and "Response B" are relatively neutral. But labels like "Expert response" or "Professional answer" can lead the judge to apply higher standards, independent of content. Similarly, "Model A" and "Model B" are neutral, but "GPT-4 response" and "baseline response" are not, because the judge has prior beliefs about GPT-4 that will influence its evaluation. Keep labels neutral and consistent, and do not reveal model identities to the judge.

Original question placement. Including the original user question immediately before each response (rather than only once at the top of the prompt) helps the judge evaluate task completion locally. This is particularly important for evaluation tasks where different responses may interpret the question differently. Seeing the question again primes the judge to ask "does this response answer what was asked?" rather than evaluating the response in a more general sense. For long prompts where the question appears many tokens before the first response, the model may not weight the question context as heavily as it should, a consequence of the attention patterns and context window dynamics we discussed in the chapters on long-context modeling.

Response length normalization. Do not truncate responses to fixed lengths before presenting them to the judge. Truncation can remove the most important part of a response and introduce artifacts that the judge interprets as quality signals. A response that appears to stop mid-sentence will score lower on coherence even if the full response was perfectly coherent. If length is a concern for prompt budget, filter out unusually long responses before evaluation rather than mid-response truncation.

Formatting artifacts. Some responses contain formatting that looks different in a plain-text prompt than it does in the rendered interface where the user would read it. Markdown headings, bullet points, and code blocks are interpreted differently by LLM judges than by human readers looking at a rendered interface. If your evaluation task involves responses that use rich formatting, consider whether your judge needs to see the raw markdown, a rendering description, or a stripped plain-text version. Inconsistent formatting handling is a source of criterion invalidity that is easy to miss.

The output format instruction tells the judge how to express its verdict. Common formats include:

• Scalar score: A numeric rating on a fixed scale (1-5, 1-10, 0-100).
• Categorical label: One of a fixed set of labels (Excellent, Good, Fair, Poor).
• Binary choice: A or B (for pairwise comparison).
• Structured JSON: A dictionary with fields for score, reasoning, and confidence.
• Chain-of-thought + verdict: Explanation followed by a final score or choice.

Each format has tradeoffs. Scalar scores are easy to aggregate but introduce anchoring effects and quantization noise. Categorical labels are easy to interpret but collapse nuance. Binary choices are easy to extract but discard magnitude information. Structured JSON outputs are flexible but increase prompt complexity and can cause parsing failures if the judge does not follow the schema precisely.

Chain-of-thought evaluation is worth the additional token cost in most situations. Studies have found that judges using chain-of-thought formats show higher agreement with human raters and greater consistency across prompt variants than judges creating direct scores. The mechanism is similar to chain-of-thought prompting for generation tasks: the explicit reasoning trace structures the model's probability distribution in a way that reduces noise and anchoring effects. When the judge must write down its reasoning before scoring, it is effectively constrained to produce a score that is consistent with the written reasoning, which reduces the influence of surface-level priming effects.

There is a practical downside to chain-of-thought evaluation: parsing. The reasoning trace and final verdict must be reliably extractable from the judge's output. If you ask for "Reasoning: [one sentence]" followed by "SCORE: [number]", the judge might add a preamble, use different formatting, or nest the score in a sentence rather than on its own line. Your extraction logic must be reliable to these variations. Regular expressions and structured parsing both work, but you need to test extraction failure rates on real judge outputs before deploying. A 5% parsing failure rate means 5% of your evaluations return no verdict, which can introduce systematic bias if parsing failures are correlated with certain response types.

Output extraction. Regardless of the format you choose, you need a reliable way to extract the judge's verdict from its response. If you request a JSON output and the judge wraps it in markdown fences or adds a preamble, your parser will fail. Design your output format instructions to be explicit about expected structure and test extraction reliability across a sample of outputs before deploying. Include a fallback parsing step for malformed outputs. A pattern that works reliably is to ask for the score on a line by itself in a distinctive format (for example, "SCORE: 4") and use a strict regex that looks for that exact pattern. More complex formats create more parsing opportunities for failure.

Assigning the judge a specific role (for example, "You are a senior software engineer reviewing code quality") can improve criterion alignment by orienting the model's prior knowledge and judgment toward the relevant domain. Role assignment works best when:

• The dimension being evaluated is domain-specific (code correctness, medical accuracy, legal reasoning).
• The judge model has been trained on substantial data from that domain.
• The role description is consistent with the evaluation criteria.

Role assignment can backfire when the assigned role conflicts with the evaluation objective or when it activates associations that introduce unintended biases. A role assignment like "You are a strict grader" may produce harsher scores across the board, not because responses are worse but because the role primes severity. Similarly, "You are a helpful assistant" can lead the judge to be more lenient than warranted, because the "helpful assistant" persona in most instruction-tuned models is trained to be affirming rather than necessary. The word "strict" carries connotations of severity that go beyond a simple instruction to be accurate.

The most subtle failure mode is domain-specific role assignments that carry implicit style preferences. Telling a judge "You are a professional academic reviewer" may cause it to penalize informal language even when the task was conversational. The role description activates expectations about writing register that are irrelevant to the quality dimension you are measuring. If you suspect this is happening, ablate the role description entirely and compare the distribution of scores. If scores shift substantially when you remove the role, the role is doing more than just domain alignment. It is influencing the implicit standard of judgment in ways that may not reflect what you want. Test your role assignments empirically by comparing judge verdicts with and without the role description before committing to them.

A useful middle ground is a role description that specifies domain knowledge without specifying a judgment style. "You are an expert in machine learning" is cleaner than "You are a strict machine learning professor" because it provides domain priming without style priming. The criteria and scale anchors you provide in the evaluation prompt should carry the judgment style, not the role description.