LLM-as-Judge: Scalable AI Evaluation with Language Models › Judge Calibration

Raw judge scores are rarely publication-ready. Different judge models use different regions of the score range, respond differently to the same rubric, and vary in their consistency. Calibration turns raw judge output into scores that correspond to human quality judgments. Without calibration, you cannot know whether a judge score of 4 means "excellent" or "mediocre" or whether the judge is measuring what you think it is.

Calibration has two components: measuring how well the judge agrees with humans, and transforming the judge's scores so that they align with the human score distribution. The first component is diagnostic. It tells you whether the judge is worth using for your task. The second component is corrective. It adjusts the judge's output to use the same scale humans would use.

The first calibration step is collecting a ground-truth dataset: a sample of responses rated by human annotators, using the same rubric the judge will apply. Typical datasets for calibration range from 200 to 2000 examples, large enough to measure correlation reliably without requiring a full annotation effort.

Choosing this calibration sample carefully is important. The sample should represent the full distribution of responses you expect to evaluate in production, not just easy or representative cases. If your evaluation includes difficult edge cases (responses that are technically correct but unhelpful, or responses that address the question well but contain subtle factual errors), those cases should appear in the calibration sample at roughly the same frequency as in production. A calibration sample biased toward easy cases produces an optimistic correlation estimate that overstates how well the judge performs on hard cases.

You then run your judge model on the same examples and compute the correlation between judge scores and human scores. Two metrics are standard:

Pearson correlation measures the linear relationship between judge scores and human scores. It answers the question: as human scores go up, do judge scores go up by a proportional amount?

$$r = \frac{\sum_{i=1}^{n} (j_i - \bar{j})(h_i - \bar{h})}{\sqrt{\sum_{i=1}^{n}(j_i - \bar{j})^2} \cdot \sqrt{\sum_{i=1}^{n}(h_i - \bar{h})^2}}$$

where:

• $j_i$: the judge score for evaluation example $i$
• $h_i$: the human annotator score for example $i$
• $\bar{j}$: the mean judge score across all $n$ examples
• $\bar{h}$: the mean human score across all $n$ examples
• $n$: the total number of calibration examples

The numerator measures how much judge and human scores co-vary. When both scores tend to be high at the same time and low at the same time, the numerator is positive and large. The denominator normalizes by each variable's spread, bounding the result between $-1$ (perfect inverse relationship) and $+1$ (perfect linear agreement). A Pearson correlation of $r = 0$ means the judge scores are linearly uncorrelated with human scores: knowing the human score tells you nothing about the judge score.

Pearson correlation assumes linear agreement and is sensitive to scale differences: if the judge systematically uses a compressed score range, Pearson penalizes that even when the rankings agree perfectly. Imagine a judge that consistently returns scores from 3 to 5 while human annotators use the full 1 to 5 range. The rankings may agree perfectly, but Pearson will be lower than it would be with aligned scales, because the judge's compressed range does not map linearly onto the human range.

Spearman rank correlation measures whether judge rankings agree with human rankings, regardless of whether the scales align. It first converts both sets of scores to ranks, then computes their correlation:

$$\rho = 1 - \frac{6 \sum_{i=1}^{n} d_i^2}{n(n^2 - 1)}$$

where:

• $d_i$: the difference in rank assigned by the judge and by humans for example $i$ (e.g., if the judge ranks example 3 as 2nd best but humans rank it 4th best, $d_3 = 2$)
• $n$: the total number of calibration examples
• The factor $6$ and the denominator $n(n^2 - 1)$ normalize the sum of squared rank differences to a $[-1, +1]$ scale

Spearman is more reliable to non-linear scale differences than Pearson and is often the preferred calibration metric for evaluation systems, because getting the rankings right matters more than getting the absolute scores right. If you are using judge scores to select the best model or flag low-quality responses, you care primarily about rank order: does the judge assign higher scores to better responses? Spearman directly measures this property.

The formula can be understood intuitively. If judge and human rankings agree perfectly, $d_i = 0$ for every example, and $\rho = 1 - 0 = 1$. If they disagree maximally (the judge ranks everything in exactly the reverse order from humans), the sum of squared rank differences is maximized, and $\rho = -1$. In practice, for a well-calibrated judge on language quality tasks, you should aim for Spearman $\rho > 0.7$.

A well-calibrated judge achieves Spearman $\rho > 0.7$ with human annotators on the relevant task type. Values below 0.5 indicate the judge is not measuring what you think it is measuring.

What do you do if calibration is poor? First, check whether the problem is the rubric or the judge model. Run the judge with a simplified rubric on a subset of examples and compare to human scores. If correlation improves substantially, the rubric is the problem: your score anchors are ambiguous or the dimensions are too abstract. If correlation stays low even with a simplified rubric, the judge model is the problem: it lacks the capability to reliably distinguish quality on your task type. In that case, switch to a more capable judge model or supplement with execution-based evaluation.

Second, check whether calibration failures cluster on specific example types. A judge that achieves $\rho = 0.8$ on easy examples but $\rho = 0.3$ on difficult examples has a systematic gap in handling hard cases, which may be acceptable if hard cases are rare in your production distribution. Stratified calibration analysis reveals these patterns that aggregate metrics obscure.

Third, check for annotation disagreement in your human scores. If your human annotators themselves disagree significantly on the calibration sample (which you can measure by computing inter-annotator agreement), then low judge-human correlation may partly reflect the noise in the human labels rather than a judge failure. Computing the judge's correlation with the average of multiple independent annotations (rather than a single annotator) gives a more stable calibration target.

Even when rank correlation is high, judge scores may occupy a different region of the scale than human scores. A judge might consistently return scores of 6-8 when human annotators span 3-9. The fix is to z-score the judge scores and then rescale them to match the human score distribution:

$$j_i^{\text{norm}} = \frac{j_i - \mu_j}{\sigma_j} \cdot \sigma_h + \mu_h$$

where:

• $j_i$: the raw judge score for example $i$
• $\mu_j$: the mean of raw judge scores across the calibration set
• $\sigma_j$: the standard deviation of raw judge scores
• $\mu_h$: the mean of human scores across the calibration set
• $\sigma_h$: the standard deviation of human scores
• $j_i^{\text{norm}}$: the normalized judge score, now expressed on the same distribution as human scores

The first term, $\frac{j_i - \mu_j}{\sigma_j}$, centers and scales the judge score to a zero-mean, unit-variance distribution. This removes the judge's systematic offset and compresses or expands the range. The second term, $\cdot \sigma_h + \mu_h$, maps it back to the human score distribution. This preserves the rank ordering while aligning the scale, so a judge that compresses scores into 6-8 will be renormalized to use the full human range of 3-9.

Normalization has clear limits. It corrects for scale shifts and compressions, but it cannot fix a judge that assigns scores randomly or that systematically misranks responses. If Spearman correlation is low, normalization will not help: you will be applying a linear transformation to a score that does not rank examples in the right order. Normalization is a post-processing step that assumes the judge's rank ordering is approximately correct and only adjusts the scale. Run calibration analysis before deciding whether normalization is the right tool.

LLM outputs are stochastic. Running the same prompt twice at temperature > 0 may produce different scores. You can reduce variance by sampling multiple independent judgments and averaging them:

$$\bar{j}_i = \frac{1}{K} \sum_{k=1}^{K} j_i^{(k)}$$

where:

• $\bar{j}_i$: the averaged judge score for example $i$
• $K$: the number of independent judge samples drawn for each example
• $j_i^{(k)}$: the score from the $k$-th sample for example $i$

This is a direct application of the law of large numbers: the average of $K$ independent samples converges toward the expected score, with variance decreasing as $\frac{\sigma^2}{K}$. In practice, $K = 3$ to $5$ provides meaningful variance reduction; beyond that, diminishing returns set in because the remaining variance reflects judge uncertainty rather than sampling noise.

There is an important subtlety in how you interpret the variance across samples. High variance across repeated judge runs on the same example signals real ambiguity: the judge cannot confidently place the response on the scale. This ambiguity is informative. Responses with high judge score variance are likely to also show high human annotator disagreement. Rather than just averaging away this variance, consider flagging high-variance examples for human review. They represent the cases where automated evaluation is least reliable and where human judgment adds the most value.

Alternatively, you can set temperature to 0 for deterministic outputs. This eliminates sample-to-sample variance but may produce outputs with less fine-grained reasoning than higher-temperature samples. Deterministic outputs are preferable in production monitoring contexts where consistency across evaluation runs is more important than nuance. For research evaluations where you want to characterize judge uncertainty, using temperature > 0 and reporting variance gives a richer picture.

In high-stakes evaluation contexts, you may want to define explicit thresholds at which judge scores trigger human review rather than automatic decision-making. For example, you might use LLM judges for routine monitoring but escalate any response with a judge score below 2 or with high variance across samples to a human reviewer. This hybrid approach captures the cost benefits of automated evaluation while preserving human oversight for the cases where it matters most.

Designing a good escalation policy requires understanding the operating point of your evaluation system. How many examples do you expect to fall below the threshold? What is the cost of false positives (escalating good responses) versus false negatives (not escalating bad responses)? You can estimate these quantities from your calibration data by simulating different threshold policies and measuring the resulting escalation rate and judge-human agreement in the escalated versus non-escalated subsets.