Evaluation Prompt Engineering: Designing Reliable LLM Judges › Prompt Sensitivity

Before you can design a good evaluation prompt, you need to appreciate how sensitive LLM judges are to prompt variation. The same two responses evaluated with slightly different prompts can produce opposite verdicts. This is not a theoretical concern. It has been demonstrated empirically across judge models, task types, and prompt modifications that span the range from trivial to substantial.

Prompt sensitivity refers to changes in judge output caused by changes in the evaluation prompt that should, in principle, be irrelevant to quality. A judge that is reliable to prompt variation should give the same verdict regardless of whether you phrase the instruction as "evaluate the quality" or "assess the quality," and regardless of whether you ask for a score from 1 to 5 or a score from 1 to 10.

In practice, LLM judges vary in their outputs across a wide range of modifications:

• Wording changes: "Rate the response on fluency" vs. "How fluent is the response?" can produce different distributions of scores, even when the underlying measurement task is identical.
• Criteria ordering: If your prompt lists accuracy, clarity, and helpfulness, the judge tends to weight criteria listed first more heavily than criteria listed last. This is a close cousin of position bias, but applied to the criteria themselves rather than to the responses.
• Instruction tone: Prompts that frame the task as "find flaws in the response" elicit harsher judgments than prompts that frame it as "assess the strengths and weaknesses." The label changes the cognitive posture of the judge.
• Format instructions: Whether you ask for a numeric score, a letter grade, or a categorical label (Excellent / Good / Fair / Poor) changes the output format and the underlying distribution of judgments. Some output formats introduce quantization artifacts that are not present in others.
• Scale anchoring: A prompt that defines the endpoints of a 1-to-5 scale ("1 = completely wrong, 5 = perfectly accurate") produces different results than a scale with no anchors or with different anchor descriptions.

Understanding these sources of sensitivity is the first step toward controlling them. Treating any one of them as fixed and unproblematic is a mistake that will show up in your evaluation results.

The sensitivity of LLM judges to prompt variation reflects properties of how language models are trained. Transformers are trained to predict the next token given all previous tokens. When an evaluation prompt includes specific wording, that wording primes the model's probability distribution over subsequent tokens. This is the same mechanism that makes few-shot prompting effective for generation tasks. For evaluation tasks, it means that the framing of the prompt actively shapes what the judge considers important.

To make this concrete: if your evaluation prompt uses the phrase "professional and polished," the judge's attention is drawn toward surface-level linguistic features associated with professional writing in its training data. That might not be what you care about. You might care about whether the response was correct, or whether it addressed the question. But the phrase "professional and polished" primes the model's internal representations toward stylistic evaluation, not accuracy evaluation, and that shift happens before the model even processes the responses being judged.

Instruction-tuned and RLHF-trained models are additionally sensitive because their training involved human feedback on specific types of prompts. If the prompts used during RLHF training tended to reward certain phrasings or structures, the judge model learns to associate those phrasings with high-quality outputs. Asking the judge to evaluate using very different language may fall outside the distribution of its fine-tuning data and produce less reliable results. This is especially pronounced for smaller judge models: a larger model with more reliable instruction-following may handle unusual prompt phrasings more gracefully, while a smaller model may exhibit more erratic behavior when the evaluation prompt deviates from familiar patterns.

There is also a subtler issue. When the same judge model is used for both generation and evaluation, the model's generation tendencies and its evaluation tendencies interact. A model trained to produce long, detailed outputs will tend to reward long, detailed outputs when acting as a judge, because that style is associated with "good responses" in its training distribution. This conflation of "responses like mine" with "good responses" is a form of self-similarity bias that shows up when generation and evaluation share a model. The sensitivity to prompt wording partially reflects this: prompts that align with the judge's generation style activate a different prior than prompts that do not.

You can quantify prompt sensitivity for any evaluation task by constructing a set of prompt variants that should, in principle, be equivalent, and measuring how much judge verdicts vary across them. The simplest approach uses Krippendorff's alpha or Cohen's kappa, which we introduced in the chapter on inter-annotator agreement, treating each prompt variant as a different "annotator."

Constructing a useful set of variants requires covering different dimensions of variation. A minimal sensitivity test should include at least three types of changes: a wording paraphrase (same meaning, different phrasing), a scale format change (for example, 1-4 vs. 1-5 vs. Excellent/Good/Fair/Poor), and a role description change (same expertise domain, different framing). If your judge produces consistent verdicts across all three types, you have good evidence that the measured verdicts reflect the responses themselves rather than the prompt's framing. If it fails on any type, you have identified which component of your prompt needs redesign.

If the agreement across prompt variants is high (alpha > 0.8), your evaluation prompt is reliable to those particular variations. If agreement is low (alpha < 0.6), you have evidence that the framing of the prompt is driving verdicts, not the quality of the responses. That is a signal to redesign the prompt before using it to compare models. Borderline cases will always show more variance than clear cases. What you are looking for is high agreement on cases you expect to be clear-cut: obviously correct responses and obviously wrong ones should get the same verdict regardless of which variant you use.

Sensitivity measurement should not be a one-time audit. Every time you modify an evaluation prompt, even for seemingly minor improvements, re-run the sensitivity test. Prompt changes that appear cosmetic can shift verdicts in unexpected ways. A good practice is to maintain a small reference set of test cases, ranging from clear-cut correct responses to clear-cut wrong ones to a few authentic borderline cases, and run every new prompt version against this reference set before deploying it. If any clear-cut case changes verdict, you have introduced a regression.