Evaluation Prompt Engineering: Designing Reliable LLM Judges › Limitations and Practical Considerations

Evaluation prompt engineering improves judge reliability and validity, but it does not solve all problems with LLM-based evaluation. Several basic limitations remain even with well-designed prompts.

Criterion validity is not guaranteed. Even if your judge reliably applies your criteria, those criteria may not measure what matters for your task. A judge designed to measure factual accuracy will give high scores to responses that are accurate but unhelpful. A judge measuring fluency will rate responses that sound confident but are wrong. The question of what to measure, and whether your criteria capture it, is a human judgment that no amount of prompt engineering can fully automate. Periodic human review of high-scoring and low-scoring judge outputs is needed to detect criterion drift. When you examine high-scoring outputs and find that they do not appear to be high-quality responses, your criteria are not capturing what you care about.

Few-shot examples encode annotator biases. The examples you include in your evaluation prompt carry the biases of whoever labeled them. If your few-shot examples come from a single annotator with strong preferences about response style, the judge will learn those preferences and amplify them. This is not a defect of few-shot evaluation as a technique, but it is a risk that scales with how opinionated your annotators are and how much you trust their judgments. Measuring inter-annotator agreement on your few-shot example labels before including them in the prompt is a practical safeguard. If two independent annotators would not agree on the score for a proposed example, that example is not a reliable anchor for the judge's calibration.

Evaluation prompts require maintenance. As the models you are evaluating improve, the distribution of responses shifts. Criteria that were adequate for an earlier generation of models may be too coarse to discriminate between responses from a stronger model. Evaluation prompts that were calibrated against one model's output distribution may be systematically mis-calibrated when applied to a different model. This is especially important when using LLM judges for model selection: if the judge was calibrated on the same model it is evaluating, or on a model with similar failure modes, it will miss the failures that matter.

Scale inflation is common. In practice, LLM judges tend to be more lenient than human raters. Without careful calibration, scores cluster at the high end of the scale, which compresses your ability to distinguish between good and great responses. Monitoring the full score distribution across your evaluation corpus, and adjusting scale anchors or few-shot examples to spread it more evenly, is an ongoing maintenance task rather than a one-time setup step.

The judge's knowledge is bounded. For factual accuracy evaluation, the judge can only assess claims that fall within its training knowledge. Responses about recent events, specialized domains, or niche topics may be evaluated incorrectly not because the evaluation prompt is poorly designed but because the judge lacks the knowledge needed to verify the claims. This is a basic limitation of LLM-based factual evaluation that cannot be solved by prompt engineering alone. For knowledge-necessary evaluation tasks, combining the LLM judge with retrieval (giving the judge with relevant reference documents) or with a specialized knowledge base check addresses this gap.

Prompts that work for one model may not transfer. An evaluation prompt that produces high agreement with human raters when using GPT-4 as the judge may produce lower agreement when using a smaller or differently aligned model. The sensitivity characteristics we discussed depend on the specific judge model's training distribution and instruction-following capabilities. Before deploying an evaluation prompt with a new judge model, re-run your sensitivity tests and human agreement measurements. Do not assume that a validated evaluation prompt transfers automatically.

The techniques in this chapter also connect directly to the upcoming chapters on bias and fairness. The biases we have discussed here, such as length bias, style bias, and annotator bias encoded in few-shot examples, have systematic effects on which models score well in your evaluation pipeline. When evaluation prompts reward verbosity, models that produce longer outputs will appear to perform better regardless of quality. When few-shot examples reflect one cultural register or writing style, models that produce text in different registers will be systematically undervalued. These effects compound across large-scale evaluations and can lead to misleading conclusions about which models are best suited for deployment.