LLM-as-Judge: Scalable AI Evaluation with Language Models › Code Implementation
Language AI HandbookAll chapters LLM-as-Judge: Scalable AI Evaluation with Language Models › Code ImplementationOne section · Sections in this chapter6 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement a complete LLM-as-Judge pipeline using standard Python tools. We'll build pointwise scoring for a set of question-answering examples, then compute correlation with simulated human annotations to demonstrate calibration.
Setup and Imports
Defining the Judge Prompt Template
The template below encodes role framing, multi-dimensional rubric, chain-of-thought reasoning, and structured output format in a single reusable string. Wrapping it in a function that accepts instruction and response parameters makes it easy to batch across an evaluation dataset.
JUDGE_PROMPT_TEMPLATE = """You are an expert evaluator assessing AI assistant responses.
[Instruction]
{instruction}
[Response to Evaluate]
{response}
Rate the response on the following dimensions using a 1-5 scale:
Instruction Following:
1 = Response ignores or misunderstands the question
3 = Response partially addresses the question
5 = Response fully and precisely addresses all parts
Factual Accuracy:
1 = Multiple clear factual errors
3 = Minor inaccuracies or ambiguities
5 = No factual errors, fully accurate
Clarity:
1 = Response is confusing or hard to follow
3 = Response is understandable but could be clearer
5 = Response is exceptionally clear and well-organized
First reason through your evaluation, then provide scores in this exact format:
<reasoning>Your step-by-step evaluation here.</reasoning>
<score_instruction_following>X</score_instruction_following>
<score_factual_accuracy>X</score_factual_accuracy>
<score_clarity>X</score_clarity>"""Parsing Judge Output
Parsing is straightforward with regular expressions because we specified structured XML-style tags in the prompt. The function below extracts per-dimension scores and the reasoning text separately, so you can log both for debugging.
def parse_judge_scores(judge_output: str) -> dict[str, int | None]:
"""Extract structured scores from judge model output."""
dimensions = ["instruction_following", "factual_accuracy", "clarity"]
scores = {}
for dim in dimensions:
pattern = rf"<score_{dim}>(\d+)</score_{dim}>"
match = re.search(pattern, judge_output)
scores[dim] = int(match.group(1)) if match else None
return scores
def parse_reasoning(judge_output: str) -> str:
"""Extract the reasoning section from judge output."""
match = re.search(r"<reasoning>(.*?)</reasoning>", judge_output, re.DOTALL)
return match.group(1).strip() if match else ""Simulated Evaluation Dataset
For this tutorial we use a simulated dataset of question-answer pairs with pre-scored judge outputs. In production you would call a real judge model API. The simulated outputs are designed to realistically represent the kinds of subtle disagreements and systematic biases that appear when running a frontier judge on general-purpose QA tasks.
# Simulated judge responses for 8 QA examples
# In production: replace with actual API calls to a judge model
simulated_judge_outputs = [
"""<reasoning>The response directly answers the question about photosynthesis with accurate information about chlorophyll, light absorption, and glucose production. It is well-organized and clear.</reasoning>
<score_instruction_following>5</score_instruction_following>
<score_factual_accuracy>5</score_factual_accuracy>
<score_clarity>5</score_clarity>""",
"""<reasoning>The response attempts to explain the French Revolution but contains several historical inaccuracies and only partially addresses the causes mentioned in the question. The writing is clear and well-structured, making the argument easy to follow despite the inaccuracies: a slight verbosity actually aids readability here.</reasoning>
<score_instruction_following>3</score_instruction_following>
<score_factual_accuracy>2</score_factual_accuracy>
<score_clarity>4</score_clarity>""",
"""<reasoning>Excellent explanation of gradient descent with a clear step-by-step description and largely correct mathematical intuition. Covers the requested aspects completely. The convergence condition described is slightly simplified but close enough to be considered accurate at this level.</reasoning>
<score_instruction_following>5</score_instruction_following>
<score_factual_accuracy>4</score_factual_accuracy>
<score_clarity>4</score_clarity>""",
"""<reasoning>The response discusses networking concepts at length and uses confident, technical-sounding language, suggesting it addresses the TCP/IP question. However, on close inspection the protocol layer details are vague and not directly on-topic.</reasoning>
<score_instruction_following>3</score_instruction_following>
<score_factual_accuracy>3</score_factual_accuracy>
<score_clarity>2</score_clarity>""",
"""<reasoning>Good explanation of supply and demand but the response does not address the specific elasticity question asked. Partially on-topic.</reasoning>
<score_instruction_following>3</score_instruction_following>
<score_factual_accuracy>4</score_factual_accuracy>
<score_clarity>4</score_clarity>""",
"""<reasoning>The response misidentifies the author of Pride and Prejudice and provides incorrect publication details. These are clear factual errors. The response is written confidently and addresses most of what was asked, suggesting strong instruction following.</reasoning>
<score_instruction_following>5</score_instruction_following>
<score_factual_accuracy>1</score_factual_accuracy>
<score_clarity>3</score_clarity>""",
"""<reasoning>Clear and accurate explanation of DNA replication with correct terminology (helicase, polymerase, replication fork). Fully addresses the question.</reasoning>
<score_instruction_following>5</score_instruction_following>
<score_factual_accuracy>5</score_factual_accuracy>
<score_clarity>5</score_clarity>""",
"""<reasoning>The response lists some climate change factors but is disorganized and skips the specific impacts asked about. The factual claims are present but thin: the judge rates the accuracy as adequate rather than good given the lack of supporting detail.</reasoning>
<score_instruction_following>3</score_instruction_following>
<score_factual_accuracy>3</score_factual_accuracy>
<score_clarity>2</score_clarity>""",
]
# Simulated human annotation scores (ground truth for calibration)
# Format: [instruction_following, factual_accuracy, clarity]
human_scores_raw = [
[5, 5, 5],
[3, 2, 3],
[5, 5, 4],
[2, 3, 2],
[3, 4, 4],
[4, 1, 3],
[5, 5, 5],
[3, 4, 2],
]Running the Judge Pipeline
Parsing the simulated outputs and organizing scores into NumPy arrays enables vectorized calibration analysis. In a production system, this block would also call the judge API and handle rate limiting, retry logic, and parse failures gracefully.
# Parse all simulated judge outputs
judge_results = []
for output in simulated_judge_outputs:
scores = parse_judge_scores(output)
reasoning = parse_reasoning(output)
judge_results.append({"scores": scores, "reasoning": reasoning})
# Organize into arrays for analysis
dimensions = ["instruction_following", "factual_accuracy", "clarity"]
judge_scores_array = np.array(
[[r["scores"][d] for d in dimensions] for r in judge_results]
)
human_scores_array = np.array(human_scores_raw)Judge Scores vs. Human Scores ======================================================= Example IF_J IF_H FA_J FA_H CL_J CL_H ------------------------------------------------------- 1 5 5 5 5 5 5 2 3 3 2 2 4 3 3 5 5 4 5 4 4 4 3 2 3 3 2 2 5 3 3 4 4 4 4 6 5 4 1 1 3 3 7 5 5 5 5 5 5 8 3 3 3 4 2 2 IF=Instruction Following, FA=Factual Accuracy, CL=Clarity _J = Judge score, _H = Human score
The table shows close but imperfect alignment between judge and human scores, a realistic outcome for a well-calibrated frontier judge. Five examples show at least one discrepancy: Example 2 (judge rates clarity +1 higher. This reflects verbosity bias toward well-structured prose), Example 3 (judge misses a subtle factual inaccuracy in the convergence description, rating factual accuracy 4 instead of 5), Example 4 (judge over-rates instruction following for a verbose but off-topic TCP/IP response), Example 6 (judge over-rates instruction following for a confidently written but factually wrong Pride and Prejudice response), and Example 8 (judge under-rates factual accuracy, penalizing thin detail more harshly than the human annotator did). These discrepancies reflect systematic biases that calibration analysis is designed to surface.
Computing Calibration Metrics
# Compute Spearman rank correlation for each dimension
calibration_results = {}
for i, dim in enumerate(dimensions):
j_scores = judge_scores_array[:, i]
h_scores = human_scores_array[:, i]
pearson_r, pearson_p = stats.pearsonr(j_scores, h_scores)
spearman_rho, spearman_p = stats.spearmanr(j_scores, h_scores)
mae = np.mean(np.abs(j_scores - h_scores))
calibration_results[dim] = {
"pearson_r": pearson_r,
"pearson_p": pearson_p,
"spearman_rho": spearman_rho,
"spearman_p": spearman_p,
"mae": mae,
}Calibration Metrics: Judge vs. Human Scores ============================================================ Dimension Pearson r Spearman ρ MAE ------------------------------------------------------------ Instruction Following 0.918 0.918 0.250 Factual Accuracy 0.952 0.938 0.250 Clarity 0.956 0.962 0.125 Average Spearman ρ: 0.939
High Spearman correlation (above 0.8) across all three dimensions indicates the judge model is reliably ranking responses in the same order as human annotators. A low mean absolute error (MAE) below 0.5 indicates the raw scores are also close in addition to the rankings. When Spearman is high but MAE is also high, score normalization is appropriate: the judge is ranking correctly but using a shifted scale.
Visualizing Score Agreement
Points clustering near the diagonal indicate strong agreement between judge and human scores. Outliers reveal cases where the judge systematically over-rates or under-rates relative to humans, which guides rubric refinement. When you see a cluster of outliers at a specific score level (for example, the judge consistently over-rates responses that score 3 with humans), the rubric definition for that level needs sharper anchoring.
Score Distribution Analysis
Understanding how judge scores are distributed helps you detect common calibration problems: score clustering (all scores near 4), bimodal distributions (only 1s and 5s), or scale mismatch (judge uses 3-5 while humans use 1-5). Distribution analysis is often more informative than aggregate correlation metrics because it shows where on the scale the disagreement is concentrated.
# Compute distribution statistics for judge and human scores
distribution_stats = {}
for i, dim in enumerate(dimensions):
j_vals = judge_scores_array[:, i].astype(float)
h_vals = human_scores_array[:, i].astype(float)
distribution_stats[dim] = {
"judge_mean": np.mean(j_vals),
"judge_std": np.std(j_vals),
"human_mean": np.mean(h_vals),
"human_std": np.std(h_vals),
}Score Distribution Statistics ================================================================= Dimension Judge Mean Judge Std Human Mean Human Std ----------------------------------------------------------------- Instruction Following 4.00 1.00 3.75 1.09 Factual Accuracy 3.38 1.32 3.62 1.41 Clarity 3.62 1.11 3.50 1.12
When judge mean and human mean diverge by more than 0.5 points, consider applying linear score normalization. When judge standard deviation is much smaller than human standard deviation, the judge is compressing scores toward the center of the scale. Score compression is one of the most common calibration failures: the judge is reluctant to assign extreme scores, so the full quality range is represented in a compressed band. Normalization stretches that band back out.
Visualizing the score distributions side by side makes these patterns immediately visible:
When judge and human distributions overlap closely, the judge is well-calibrated. When the judge distribution is narrower (fewer extreme scores), it is compressing toward the center. When it is shifted (higher or lower on average), linear normalization can correct the offset.
Key Parameters
The key parameters controlling LLM-as-Judge pipeline behavior are:
dimensions: The evaluation criteria the judge scores. More dimensions give richer signal but require a longer prompt and more rubric design effort.K(number of samples): How many independent judge outputs to average per example. Higher values reduce variance at a linear cost increase. Values of 3 to 5 work well in practice.- Temperature: Controls output stochasticity. Temperature 0 gives deterministic scores; temperature 0.7 gives more fine-grained reasoning but higher variance across runs.
- Calibration set size: The number of human-annotated examples used to compute correlation metrics. Aim for 200 to 2000 examples; fewer examples make correlation estimates unreliable.
- Score range: The numeric scale for pointwise ratings (e.g., 1-5 or 1-10). Narrower scales (1-5) are easier for judges to apply consistently; wider scales allow finer discrimination.