Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LLM-as-Judge: Scalable AI Evaluation with Language Models › Worked Example: Designing and Running an LLM Judge

Michael BrenndoerferMarch 10, 20262 min read

Part 55 of 67Human and Model Evaluation

Chapter 426LLM-as-Judge: Scalable AI Evaluation with Language Models › Worked Example: Designing and Running an LLM Judge

Reading section 5 of 10 · Worked Example: Designing and Running an LLM Judge · 2 minSection 5 of 10 · 2 min

Section 5 of 10 · 2 min read

LLM-as-Judge: Scalable AI Evaluation with Language Models › Worked Example: Designing and Running an LLM JudgeOne section · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's walk through a concrete evaluation scenario to make these concepts concrete. Suppose you are building a customer support chatbot and want to evaluate response quality on a set of test queries.

Evaluation goal: Score each response on instruction following and helpfulness, on a scale of 1 to 5 each.

Judge model: GPT-4 or equivalent frontier model.

Dataset: 50 customer queries with corresponding chatbot responses.

Judge prompt:

You are an expert evaluator of customer support AI assistants.

[Customer Query]
{query}

[Chatbot Response]
{response}

Rate the response on the following criteria:

Instruction Following (1-5):
  1 = Response ignores the question entirely
  3 = Response partially addresses the question
  5 = Response fully addresses all aspects of the question

Helpfulness (1-5):
  1 = Response is useless or actively harmful
  3 = Response provides some value but misses key information
  5 = Response is maximally helpful and complete

Think step by step, then provide your ratings:
<reasoning>Your reasoning here.</reasoning>
<score_instruction_following>X</score_instruction_following>
<score_helpfulness>X</score_helpfulness>

Analysis workflow:

After running the judge, you parse scores from the structured output tags and compute per-dimension averages across all 50 examples. Track score distributions to detect systematic issues. If 80% of helpfulness scores cluster at 3, the rubric may need refinement (the anchor for 3 is too broadly appealing) or the model may have a real gap in giving complete answers. If scores cluster at the top, the rubric's definition of 5 may be too easy to satisfy.

The worked example illustrates why rubric design is iterative. You rarely write a rubric once and get it right. The first round of judge outputs often reveals that score anchors are ambiguous, that certain response types fall into gaps between described levels, or that the judge is interpreting a dimension differently than you intended. The calibration cycle, of running the judge, checking against a sample of human-reviewed examples, and revising the rubric, is where most of the quality improvement in LLM-as-Judge pipelines comes from.