Position Bias in LLM Judges › Worked Example: Observing and Measuring Bias

Let's walk through a concrete scenario to see how these biases manifest in practice. Consider a question from a code explanation benchmark: "Explain the difference between a list and a tuple in Python." Two responses have been generated.

Response A (short, accurate): "Lists are mutable sequences that can be changed after creation (append, remove, etc.). Tuples are immutable sequences that cannot be modified. Use tuples for fixed collections of items, lists when you need to add or remove elements."

Response B (long, padded): "Great question! Python offers several sequence types, each with unique characteristics. Lists are one of Python's most versatile data structures. They are mutable, meaning you can add, remove, and modify elements after the list is created. You can use methods like append(), extend(), remove(), and pop() to modify a list. Tuples, on the other hand, are immutable. Once created, you cannot change their contents. They are often used for fixed collections of data, like database records or coordinate pairs. In summary, use lists when mutability is needed, tuples for fixed data. Both support indexing and slicing."

A human expert rating these on quality would likely rate them roughly equal. Both are accurate and cover the core distinction. Response B adds padding (the opening "Great question!", repetitive phrasing, excess detail that does not add clarity), but it is not inaccurate. A well-calibrated judge should rate them similarly or perhaps slightly prefer the concise Response A for its directness and signal-to-noise ratio.

Now we run the experiment. We ask the same judge model the same question in both orders.

Evaluation 1 (A first): The judge sees Response A first, then Response B. It rates Response B as better: "Response B provides a more complete explanation with practical method examples, which is helpful for understanding how to apply these sequence types."

Evaluation 2 (B first): The judge sees Response B first, then Response A. It again rates Response B as better: "Response B provides a thorough, well-structured explanation that covers multiple aspects of the comparison."

The verdict is consistent but for the wrong reasons. The judge preferred B in both orderings, but this is verbosity bias, not honest quality assessment. The longer response won regardless of order. If we ran this experiment across 100 similar pairs where quality and length point in opposite directions, we would find that length predicts the judge's winner better than human quality ratings in a meaningful fraction of cases.

Now let's modify the scenario to isolate position bias. Suppose we have two responses of identical length and structure, and human raters assign them equal scores. We run the swap experiment:

Evaluation 1 (A first): Judge picks A. "Response A directly addresses the key differences in a clear, organized manner."

Evaluation 2 (B first): Judge picks B (which is now in position A). "Response B directly addresses the key differences in a clear, organized manner."

The rationale given is nearly identical, but the winner flipped. The judge always picked the first response. This is pure position bias, completely independent of content. The rationale it generates is post-hoc justification for a position-driven decision, and the near-identical language in both verdicts reveals that the judge did not engage differently with the two responses in the two orderings.

This worked example illustrates an important diagnostic: track which response wins and the language the judge uses to justify its verdict. When swap-inconsistent judges produce nearly identical rationales regardless of which response won, the judge's reasoning is following the bias rather than generating it.