Evaluation Prompt Engineering: Designing Reliable LLM Judges › Evaluation Consistency

Even a well-designed evaluation prompt will show some variance. You run the same judge on the same pair of responses twice and get different scores. This is expected: autoregressive language models are stochastic by default, and the temperature settings that produce fluent, natural outputs also introduce randomness in evaluation decisions. Managing evaluation consistency means understanding the sources of variance, quantifying them, and applying strategies to reduce variance where it matters.

Variance in LLM judge outputs comes from three main sources.

Sampling randomness. With temperature > 0, the judge samples from a probability distribution over tokens. On borderline cases where the judge's uncertainty is high (the probability mass is spread across multiple possible scores), different samples from the same distribution will produce different verdicts. Setting temperature to 0 eliminates sampling variance but introduces its own artifacts: greedy decoding can get stuck in degenerate outputs, and it removes the ability to estimate uncertainty by sampling multiple times.

Prompt sensitivity variance. As we discussed earlier, small changes to the evaluation prompt produce different verdicts. Even without changing the prompt intentionally, environmental factors like token truncation at the context window boundary or subtle differences in how the system prompt is formatted can introduce variation.

Positional and contextual effects. The context of other responses in a batch evaluation, the order in which responses appear in a multi-turn conversation, and the recent history of the judge's outputs can all influence individual verdicts. These effects are less understood than sampling variance but are real. When evaluating many responses in sequence, there is evidence that judge behavior can drift as the conversation history accumulates, particularly if the judge is creating chain-of-thought reasoning that fills the context window with its own prior verdicts.

The standard approach to measuring evaluation consistency is to run the judge multiple times on the same inputs and compute agreement metrics across runs. The agreement metric you use depends on your output format:

• Pairwise comparisons (A vs. B): Use percent agreement or Cohen's kappa across runs.
• Scalar scores: Use intraclass correlation coefficient (ICC) or Pearson correlation across runs.
• Categorical labels: Use Fleiss' kappa treating each run as a separate "rater."

A well-designed evaluation prompt at temperature 0 should show near-perfect consistency on clear cases (> 95% agreement across runs) and measurable but limited variance on borderline cases (70-85% agreement). If you see substantial variance even on clear cases, the evaluation prompt needs revision. Variance on clear cases indicates that the prompt's criterion definitions are not anchoring the judge's interpretation effectively, or that the output format is introducing parsing ambiguity.

Several strategies reliably improve evaluation consistency.

Reduce temperature. Setting temperature to 0 eliminates sampling variance. For evaluation tasks where you want a deterministic verdict, zero temperature is the default choice. Reserve higher temperatures for uncertainty estimation via sampling.

Sample and aggregate. For high-stakes evaluations, run the judge 5-10 times on the same input and aggregate the scores (by mean for scalar scores, by majority vote for categorical labels). This reduces the variance of the final estimate at the cost of additional token usage. The standard error of the mean for $N$ independent samples is:

$$\text{SE} = \frac{\sigma}{\sqrt{N}}$$

where:

• $\sigma$: the standard deviation of judge scores across individual runs (how spread out single verdicts are)
• $N$: the number of independent samples (judge runs) you aggregate

Doubling the number of samples reduces the standard error by a factor of $\sqrt{2} \approx 1.41$. Moving from one sample to five samples reduces it by $\sqrt{5} \approx 2.24\times$. Diminishing returns set in beyond about 10 samples for most evaluation tasks, making 3-5 samples a practical default for high-stakes comparisons.

Use structured output formats. Judges that produce structured JSON outputs or chain-of-thought explanations before the final score tend to show lower variance than judges that produce direct scores. The structure constrains the probability distribution over outputs and reduces the influence of random sampling on the final verdict.

Calibrate against held-out examples. Reserve a held-out set of (response, human verdict) pairs and measure your judge's calibration on this set periodically. Calibration drift, where the judge's score distribution shifts over time or across prompt variants, is an early signal that consistency is degrading.

Normalize the scale. If your judge is creating a distribution of scores that is biased toward one end of the scale (clustering at 4-5 on a 1-5 scale, for example), your criterion descriptions may be poorly calibrated. Revise the endpoint anchors to spread the distribution across the scale and retest. A judge that never assigns score 1 or score 2 on a 5-point scale is effectively operating on a 3-point scale, which reduces its discriminative power substantially.

Control the evaluation environment. Batch evaluations where the judge processes many items in sequence can show within-batch drift due to context accumulation. Running each evaluation in a fresh context (a new API call with no shared history) eliminates this source of variation. The tradeoff is cost: fresh contexts are more expensive than batched evaluations. For large-scale evaluation tasks, a practical approach is to use batched evaluation for preliminary ranking and fresh contexts for high-stakes comparisons.