Position Bias in LLM Judges › Code Implementation
Language AI HandbookAll chapters Position Bias in LLM Judges › Code ImplementationOne section · Sections in this chapter5 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's build a complete measurement framework for position and verbosity bias. We will create a toy evaluation dataset, run simulated pairwise comparisons with a biased mock judge, and compute the key metrics. In a real pipeline, the mock judge would be replaced by LLM API calls, but the measurement structure is identical.
First, let's set up imports and fix random seeds for reproducibility:
import random
import numpy as np
random.seed(42)
np.random.seed(42)We define data structures for our evaluation framework. The Response class tracks both text content and the true quality score, which in a real evaluation would come from human annotation. The judge never sees the quality score directly:
from dataclasses import dataclass, field
@dataclass
class Response:
"""A model response with metadata."""
text: str
quality: float # True quality score, 0-1 (unknown to judge)
length: int = field(init=False)
def __post_init__(self):
self.length = len(self.text.split())
@dataclass
class ComparisonPair:
"""A pair of responses with an associated question."""
question: str
response_a: Response
response_b: Response
true_winner: str # 'A' or 'B' based on true qualityNow let's create a toy dataset of comparison pairs. We deliberately include cases where longer responses are lower quality, which lets us test whether the judge incorrectly rewards length over accuracy:
from typing import List
def make_toy_dataset(n_pairs: int = 80) -> List[ComparisonPair]:
"""Create evaluation pairs with controlled quality and length."""
questions = [
"What is gradient descent?",
"Explain regularization.",
"What is overfitting?",
"How does backpropagation work?",
"What is a transformer?",
]
pairs = []
for i in range(n_pairs):
q = questions[i % len(questions)]
# Assign quality randomly, independent of length
q_a = round(random.uniform(0.3, 1.0), 2)
q_b = round(random.uniform(0.3, 1.0), 2)
# Assign lengths with some correlation to quality but not perfect
# ~30% of pairs: longer response is lower quality (to detect verbosity bias)
len_a = random.randint(20, 50)
len_b = random.randint(20, 50)
if i % 3 == 0:
# Make B longer but lower quality than A
if q_a > q_b:
len_b = random.randint(80, 150)
else:
len_a = random.randint(80, 150)
text_a = " ".join([f"word{j}" for j in range(len_a)])
text_b = " ".join([f"word{j}" for j in range(len_b)])
resp_a = Response(text=text_a, quality=q_a)
resp_b = Response(text=text_b, quality=q_b)
true_winner = "A" if q_a > q_b else "B"
pairs.append(ComparisonPair(q, resp_a, resp_b, true_winner))
return pairs
dataset = make_toy_dataset(n_pairs=80)Now we implement a mock judge with tunable biases. The judge combines the true quality signal with a position bias term (which overrides quality with some probability) and a verbosity bias term (which adds a length bonus to each response's effective score):
def biased_judge(
question: str,
response_a: Response,
response_b: Response,
position_bias_strength: float = 0.3,
verbosity_bias_strength: float = 0.25,
noise: float = 0.1,
) -> str:
"""
Simulate a biased judge that combines quality, position, and verbosity signals.
Returns 'A' or 'B' as the winner.
position_bias_strength: probability of flipping to first-position regardless of quality.
verbosity_bias_strength: extra score weight given to longer responses.
noise: random noise added to simulate stochastic judge behavior.
"""
# True quality scores
score_a = response_a.quality
score_b = response_b.quality
# Add verbosity bias: longer responses get a bonus proportional to their log-length share
len_ratio_a = np.log1p(response_a.length) / (
np.log1p(response_a.length) + np.log1p(response_b.length)
)
len_ratio_b = 1 - len_ratio_a
score_a += verbosity_bias_strength * len_ratio_a
score_b += verbosity_bias_strength * len_ratio_b
# Add random noise
score_a += random.gauss(0, noise)
score_b += random.gauss(0, noise)
# Quality-based verdict
quality_verdict = "A" if score_a > score_b else "B"
# Apply position bias: with probability position_bias_strength, override to first position
if random.random() < position_bias_strength:
return "A" # A is always first in this call's perspective
return quality_verdictNow we run the swap experiment. For each pair, we evaluate in both orderings and compare verdicts. The relabeling step in the swapped ordering is important: when the judge returns "A" in the swapped call, "A" corresponds to the original Response B, so we translate the verdict back to original labels:
def run_swap_experiment(
pairs: List[ComparisonPair], judge_fn, **judge_kwargs
) -> dict:
"""
Run pairwise evaluation in both orders and compute bias metrics.
Returns a dict with all key metrics.
"""
verdicts_original = [] # verdict when A is first
verdicts_swapped = [] # verdict when B is first (relabeled to original A/B)
true_winners = []
for pair in pairs:
# Original order: A first
v1 = judge_fn(
pair.question, pair.response_a, pair.response_b, **judge_kwargs
)
# Swapped order: B first; judge returns 'A' or 'B' from its perspective,
# where the new 'A' is the original B
v2_raw = judge_fn(
pair.question, pair.response_b, pair.response_a, **judge_kwargs
)
# Re-label: if judge picked 'A' in swapped order, that means original B won
v2 = "B" if v2_raw == "A" else "A"
verdicts_original.append(v1)
verdicts_swapped.append(v2)
true_winners.append(pair.true_winner)
# Swap consistency: fraction where both orderings agree
consistent = [
v1 == v2 for v1, v2 in zip(verdicts_original, verdicts_swapped)
]
swap_consistency = np.mean(consistent)
# Position preference rate: how often position-A wins in original order
ppr_a = np.mean([v == "A" for v in verdicts_original])
# Accuracy against true winner (using original order verdict)
accuracy = np.mean(
[v == tw for v, tw in zip(verdicts_original, true_winners)]
)
# First-position bias among inconsistent pairs
inconsistent_mask = [not c for c in consistent]
inconsistent_original = [
v for v, m in zip(verdicts_original, inconsistent_mask) if m
]
first_position_bias = (
np.mean([v == "A" for v in inconsistent_original])
if inconsistent_original
else 0.5
)
return {
"swap_consistency": swap_consistency,
"position_preference_rate_A": ppr_a,
"accuracy_vs_true": accuracy,
"first_position_bias_rate": first_position_bias,
"n_inconsistent": sum(inconsistent_mask),
"n_pairs": len(pairs),
}
results_biased = run_swap_experiment(
dataset,
biased_judge,
position_bias_strength=0.3,
verbosity_bias_strength=0.25,
noise=0.1,
)
results_ideal = run_swap_experiment(
dataset,
biased_judge,
position_bias_strength=0.0,
verbosity_bias_strength=0.0,
noise=0.05,
)=== Biased Judge === swap_consistency: 0.550 position_preference_rate_A: 0.688 accuracy_vs_true: 0.762 first_position_bias_rate: 0.806 n_inconsistent: 36 n_pairs: 80 === Near-Ideal Judge (no position/verbosity bias) === swap_consistency: 0.825 position_preference_rate_A: 0.613 accuracy_vs_true: 0.887 first_position_bias_rate: 0.429 n_inconsistent: 14 n_pairs: 80
The biased judge shows swap consistency well below 1.0, confirming that verdict order affects outcomes. The ideal judge achieves near-perfect consistency, with accuracy tracking true quality closely. The first-position bias rate above 0.5 in the biased condition confirms the directional preference for first-presented responses.
Now let's measure verbosity bias by computing the correlation between length and judge score, separately from quality:
def measure_verbosity_bias(
pairs: List[ComparisonPair], judge_fn, **judge_kwargs
) -> dict:
"""
Measure how strongly response length predicts judge verdicts
after controlling for true quality.
"""
quality_advantages = [] # quality_a - quality_b
length_advantages = [] # log(len_a) - log(len_b)
judge_chose_a = []
for pair in pairs:
q_diff = pair.response_a.quality - pair.response_b.quality
l_diff = np.log1p(pair.response_a.length) - np.log1p(
pair.response_b.length
)
verdict = judge_fn(
pair.question, pair.response_a, pair.response_b, **judge_kwargs
)
chose_a = 1 if verdict == "A" else 0
quality_advantages.append(q_diff)
length_advantages.append(l_diff)
judge_chose_a.append(chose_a)
quality_advantages = np.array(quality_advantages)
length_advantages = np.array(length_advantages)
judge_chose_a = np.array(judge_chose_a)
# Compute correlation of each predictor with judge choice
corr_quality = np.corrcoef(quality_advantages, judge_chose_a)[0, 1]
corr_length = np.corrcoef(length_advantages, judge_chose_a)[0, 1]
# Compute accuracy of quality-based predictor alone
quality_predicted = (quality_advantages > 0).astype(int)
quality_accuracy = np.mean(quality_predicted == judge_chose_a)
# Verbosity bias: fraction of pairs where length and quality disagree on winner,
# and the judge picks the longer (lower-quality) response
disagree_mask = (quality_advantages > 0.05) & (length_advantages < 0)
if disagree_mask.sum() > 0:
verbosity_bias_rate = np.mean(judge_chose_a[disagree_mask] == 0)
else:
verbosity_bias_rate = 0.0
return {
"corr_quality_with_choice": corr_quality,
"corr_length_with_choice": corr_length,
"quality_prediction_accuracy": quality_accuracy,
"verbosity_bias_rate": verbosity_bias_rate,
"n_quality_length_conflict": int(disagree_mask.sum()),
}
verbosity_biased = measure_verbosity_bias(
dataset,
biased_judge,
position_bias_strength=0.0,
verbosity_bias_strength=0.4,
noise=0.05,
)
verbosity_unbiased = measure_verbosity_bias(
dataset,
biased_judge,
position_bias_strength=0.0,
verbosity_bias_strength=0.0,
noise=0.05,
)=== Verbosity-Biased Judge === corr_quality_with_choice: 0.690 corr_length_with_choice: -0.328 quality_prediction_accuracy: 0.887 verbosity_bias_rate: 0.103 n_quality_length_conflict: 29 === Unbiased Judge === corr_quality_with_choice: 0.721 corr_length_with_choice: -0.425 quality_prediction_accuracy: 0.838 verbosity_bias_rate: 0.034 n_quality_length_conflict: 29
In the verbosity-biased condition, the correlation between length advantage and judge choice is clearly positive, even in cases where the longer response is the lower-quality one. The verbosity bias rate quantifies this directly: when quality and length point in opposite directions, how often does the judge pick length over quality?
Bias Mitigation Strategies
The most widely used mitigation for position bias is answer swapping: run every comparison twice, once in each order, and aggregate the results. If the judge produces consistent verdicts across both orderings, that verdict is reliable. If the verdicts conflict, you have several options for handling the disagreement:
- Declare a tie: When verdicts differ across orderings, record no winner. This is conservative but avoids position-contaminated decisions.
- Average scores: If you use continuous scores rather than binary preferences, average the scores from both orderings. This is the most principled approach when scoring is available.
- Use structured output: Ask the judge to output an independent score for each response (such as 1 through 10) rather than picking a winner by comparison. This avoids the comparison framing that position bias exploits, though it introduces its own calibration challenges.
- Confidence-weighted aggregation: Weight the verdict from each ordering by the judge's expressed confidence. If the judge is highly confident in both orderings and they agree, use the shared verdict. If it is highly confident in one ordering but uncertain in the other, use the confident verdict.
For verbosity bias, mitigation is harder because it is embedded in the model's learned representations rather than just the prompt framing. Several approaches are used in practice:
- Normalized scoring: Ask the judge to score responses per unit of information rather than in absolute terms, or explicitly instruct it to penalize unnecessary length. This helps but requires the model to understand "unnecessary length," which is itself a judgment call.
- Length-controlled pairs: When comparing responses, truncate all responses to the same length before evaluation. This is extreme but effective for isolating quality from length effects.
- Post-hoc length penalty: After collecting judge scores, run the verbosity regression to estimate and subtract the estimated length contribution from each score. This requires a calibration dataset with human quality annotations.
- Chain-of-thought rubric evaluation: Ask the judge to evaluate each response against explicit criteria before deciding. If each criterion is independently scored, verbosity has fewer opportunities to inflate the aggregate.
For sycophancy, mitigation is the most technically challenging because it requires either using judge models maximally dissimilar from the models being evaluated, or using multiple judges and looking for consensus. Neither approach is fully reliable, but both reduce the problem. When using multiple judges, a disagreement between judges from different model families is itself informative: it suggests that the verdict may be driven by family-specific preferences rather than objective quality.
def mitigated_judge(
question: str,
response_a: Response,
response_b: Response,
judge_fn,
aggregation: str = "consistency",
**judge_kwargs,
) -> str:
"""
Run swapped evaluation and aggregate verdicts to reduce position bias.
aggregation options:
'consistency': return verdict only if both orderings agree, else 'TIE'
'majority': use first verdict when tie-broken by original order
"""
# Original order
v1 = judge_fn(question, response_a, response_b, **judge_kwargs)
# Swapped order (relabeled back to A/B)
v2_raw = judge_fn(question, response_b, response_a, **judge_kwargs)
v2 = "B" if v2_raw == "A" else "A"
if v1 == v2:
return v1
if aggregation == "consistency":
return "TIE"
else:
return v1 # default to original order on disagreement
def evaluate_with_mitigation(
pairs: List[ComparisonPair], judge_fn, **judge_kwargs
) -> dict:
"""Compare mitigated vs unmitigated evaluation accuracy."""
unmitigated_correct = 0
mitigated_correct = 0
mitigated_ties = 0
for pair in pairs:
# Unmitigated
v_unmitigated = judge_fn(
pair.question, pair.response_a, pair.response_b, **judge_kwargs
)
if v_unmitigated == pair.true_winner:
unmitigated_correct += 1
# Mitigated
v_mitigated = mitigated_judge(
pair.question,
pair.response_a,
pair.response_b,
judge_fn,
**judge_kwargs,
)
if v_mitigated == pair.true_winner:
mitigated_correct += 1
if v_mitigated == "TIE":
mitigated_ties += 1
n = len(pairs)
return {
"unmitigated_accuracy": unmitigated_correct / n,
"mitigated_accuracy": mitigated_correct / (n - mitigated_ties)
if (n - mitigated_ties) > 0
else 0,
"tie_rate": mitigated_ties / n,
"n_pairs": n,
}
mitigation_results = evaluate_with_mitigation(
dataset,
biased_judge,
position_bias_strength=0.25,
verbosity_bias_strength=0.15,
noise=0.1,
)=== Mitigation Results === unmitigated_accuracy: 0.800 mitigated_accuracy: 0.878 tie_rate: 0.487 n_pairs: 80 Interpretation: Accuracy gain from mitigation: 0.078 Cost: 48.8% of pairs declared ties (require human review)
Answer swapping improves accuracy among pairs where the judge can reach a consistent verdict. The cost is that a fraction of comparisons result in ties and require fallback handling, typically human annotation or a second judge from a different model family.
Key Parameters
The key parameters for bias measurement and mitigation are:
- position_bias_strength: The probability that a judge overrides quality with a position-based decision. Measured as 1 minus the swap consistency rate, adjusted for the direction of inconsistencies.
- verbosity_bias_strength: The regression coefficient on log-length in a quality-controlled model, quantifying how much length contributes to judge scores beyond actual content quality.
- swap_consistency: The fraction of comparison pairs where the judge produces the same verdict in both orderings. This is the primary diagnostic metric for position bias.
- first_position_bias_rate: Among inconsistent comparisons, the fraction where the judge chose the first-presented response. Values near 0.5 indicate undirected noise; values far from 0.5 indicate systematic positional preference.