Evaluation Prompt Engineering: Designing Reliable LLM Judges › Few-Shot vs. Zero-Shot Evaluation

One of the most consequential design decisions in evaluation prompt engineering is whether to include example evaluations in the prompt (few-shot) or to rely on the judge's training to interpret your criteria without examples (zero-shot). Both approaches have legitimate use cases, and the choice should be based on the specific task, the judge model, and the resources available to create high-quality examples.

Zero-shot evaluation relies entirely on the criterion descriptions and task framing in the prompt. It requires no example verdicts. Its key advantages are simplicity and generality: you do not need to curate labeled examples, and the prompt stays compact. Zero-shot evaluation works reasonably well when:

• The evaluation dimension is common in the judge's training distribution (fluency, coherence, grammaticality).
• Your criterion descriptions are precise and anchor the endpoints of the scale.
• You are evaluating a task type that is well-represented in instruction-following data.

The main risk with zero-shot evaluation is inconsistency in how the judge interprets abstract criteria. "Accuracy" might mean different things to the judge across different runs if the context does not pin it down clearly. For dimensions that are less common or more subjective, zero-shot evaluation tends to show high variance and low agreement with human raters.

Zero-shot evaluation is a reasonable starting point when you are exploring a new evaluation task and do not yet have labeled examples to draw on. You can iterate quickly on criterion descriptions without the overhead of maintaining an example set. Once you have a stable criterion definition that produces acceptable zero-shot performance, you can consider whether adding few-shot examples would improve calibration further. In many cases, a well-designed zero-shot prompt with explicit scale anchors comes close to the performance of a few-shot prompt with poor examples, which means the ordering should be: design a good zero-shot prompt first, then add examples only if you can curate high-quality ones.

Few-shot evaluation includes two to five example evaluations in the prompt, each consisting of a (question, response, verdict) triple. The examples serve as in-context demonstrations of how you want the judge to apply the criteria. Few-shot evaluation is more reliable than zero-shot evaluation on most evaluation tasks when the examples are high quality.

The benefits of few-shot evaluation are:

• Calibration: Examples anchor the judge's internal scale to your intended distribution of verdicts. If you provide one example of a score-1 response and one example of a score-5 response, the judge interpolates between them rather than relying on its default interpretation of the scale.
• Edge case coverage: Including an example that covers a common edge case teaches the judge how to handle it consistently.
• Reduced sensitivity: Judges evaluated with few-shot prompts show less sensitivity to phrasing variations in the criterion descriptions because the examples constrain interpretation more tightly than text alone.

The calibration benefit deserves elaboration. Consider a 1-to-5 scale for measuring response helpfulness. Without examples, the judge must interpret what "helpful enough to deserve a 4" means on its own, and that interpretation varies across runs because the model is sampling from a distribution. With one example of a score-4 response in the prompt, the judge has a concrete anchor: it can compare each new response against the example and decide whether it is more or less helpful. This comparison is a much more constrained task than free-form interpretation of "a 4 on a 1-5 scale." The result is lower variance and better calibration to your intended scale.

The benefit of few-shot evaluation depends entirely on the quality of the examples. A single high-quality example that represents a realistic borderline case is more useful than five examples that cluster at the extremes of the scale. Good few-shot examples share several properties:

• They cover the task types that will appear in your evaluation data (do not use toy examples for a production evaluation).
• They include a reasoning trace explaining why the response received that score, rather than only the score.
• They span the scale, including at least one clearly high-quality response, one clearly low-quality response, and one borderline case.
• They were labeled by domain experts whose judgment you trust, ideally with cross-labeler agreement measured before inclusion.

Low-quality examples introduce their own biases. If your examples systematically reflect the preferences of a single annotator or a narrow slice of task types, few-shot evaluation will amplify those biases rather than calibrate them away. An evaluation prompt with poorly chosen examples can perform worse than zero-shot evaluation with good criteria, because the examples actively mislead the judge about your intentions.

The borderline case example deserves special emphasis. Examples at the extremes (clearly correct, clearly wrong) help calibrate the scale endpoints but provide little guidance for the cases where evaluation matters: the responses where a well-designed prompt would give a 2 or 3 and a poorly designed prompt would give a 4 or 5. A borderline example with explicit reasoning explains to the judge how to handle ambiguity, which is where most evaluation value is created. If you can include only one example in a token-constrained environment, make it a borderline case.

A common failure mode in few-shot evaluation is inconsistency between the examples and the criterion descriptions. If your text says "accuracy means only verifiable claims" but your examples reward confident-sounding responses regardless of verifiability, the judge will learn from the examples and ignore the text. The examples take precedence over the prose when the two conflict because they carry more direct evidence about your intentions.

Before deploying a few-shot evaluation prompt, verify that every example is consistent with your criterion definitions. This is easier to check programmatically: use a separate judge to evaluate whether each example verdict follows from the criteria, and flag any inconsistencies for human review. You can also check consistency by having a second annotator who did not create the examples verify that they would assign the same verdicts given the criterion definitions. Systematic disagreement between the example creator and the verifier is a signal that the examples encode idiosyncratic preferences rather than your stated criteria.

The practical guidance for the number of examples is two to five, but the right number depends on the task complexity and the judge model's capability. For simple, well-defined criteria (factual correctness on closed-domain questions), two examples covering clear-correct and clear-wrong cases may be sufficient. For complex, criteria with several dimensions (overall response quality for open-ended questions), five examples that cover a range of cases and failure modes provide better calibration.

Adding more than five examples has diminishing returns and carries a cost. Each example adds tokens to the prompt, which increases latency and cost per evaluation. In long prompts, there is also a risk of context window crowding: when the examples take up a large fraction of the context window, the judge may weight the examples less carefully because they are further from the end of the prompt where the model's attention concentrates. Keep your example set small and high-quality rather than large and complete.