Position Bias in LLM Judges › The Three Biases: Position, Verbosity, and Sycophancy

These three biases form a cluster that often co-occur and interact. Understanding each individually is necessary before examining how they compound. This section digs into the mechanics of each bias, how to define it precisely, how to quantify it, and what drives it.

Position bias is a structural problem. The prompt template for pairwise evaluation typically looks like this: the question, then Response A, then Response B, then the instruction to judge. This fixed structure means that whichever response occupies position A always appears first. If the judge model has even a slight preference for first-position responses, and most do to some degree, this creates a systematic advantage that accumulates across many evaluations. At scale, even a 5 to 10 percent bias rate can substantially distort a leaderboard.

The degree of bias varies by model family and size. Smaller models tend to show stronger and less consistent position biases because they have less reliable reasoning capacity: they rely more on shallow heuristics like "the first thing mentioned was more relevant" rather than careful comparison. Larger, instruction-tuned models show reduced but non-zero position effects. Models with longer context windows may exhibit different patterns because the distance between the two responses grows as prompt length increases, changing which response is more salient at decision time.

Measuring position bias requires running evaluations in both orders. For each question, you submit the pair as (Response A, Response B) and also as (Response B, Response A), then compare the verdicts. The key metrics are as follows.

Swap consistency rate: The fraction of comparisons where the judge gives a consistent verdict regardless of order. Formally, for a set of $N$ comparison pairs, let $v_1^{(i)}$ be the verdict when pair $i$ has response $A$ first, and let $v_2^{(i)}$ be the verdict when $B$ is first (with labels relabeled to match original A and B identity). The swap consistency rate is:

$$\text{SC} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\left[v_1^{(i)} = v_2^{(i)}\right]$$

where:

• $N$: the total number of comparison pairs evaluated
• $v_1^{(i)}$: the verdict for pair $i$ when response $A$ appears first in the prompt
• $v_2^{(i)}$: the verdict for pair $i$ when the same responses are swapped, so $B$ appears first (relabeled to preserve original A/B identity)
• $\mathbf{1}[\cdot]$: the indicator function, returning 1 if the condition inside is true and 0 otherwise

A perfectly consistent judge scores SC = 1.0. Values around 0.7 to 0.8 are commonly observed in practice, meaning 20 to 30 percent of judgments flip based solely on presentation order.

Position preference rate: Beyond consistency, you want to know which direction the bias flows. The position-A preference rate counts how often the judge favors whichever response appears in position A, regardless of that response's actual quality:

$$\text{PPR}(A) = \frac{\text{count of times position-A is preferred}}{N}$$

where the numerator counts every comparison in which the judge selected the response occupying the first (A) slot as the winner, and $N$ is the total number of comparisons evaluated.

In an unbiased judge, $\text{PPR}(A) = 0.5$. Systematic deviations reveal directional position bias. If $\text{PPR}(A) = 0.65$, the judge prefers the first response 65 percent of the time, a significant structural advantage that inflates the performance of whatever model happens to be placed in position A.

Inconsistency decomposition: When a judge is inconsistent across orderings, you can decompose the inconsistencies by direction. Let $n_{A \to A}$ be the count of cases where the judge preferred position A in both orderings (i.e., preferred the first-presented response each time, regardless of which actual response that was). Let $n_{B \to B}$ be the reverse, where the judge consistently preferred the last-presented response. The first-position bias rate among inconsistent pairs is:

$$\text{FPB} = \frac{n_{A \to A}}{n_{A \to A} + n_{B \to B}}$$

where:

• $n_{A \to A}$: count of inconsistent pairs where the judge picked position A in both orderings, always choosing the first-presented response
• $n_{B \to B}$: count of inconsistent pairs where the judge picked position B in both orderings, always choosing the last-presented response
• $n_{A \to A} + n_{B \to B}$: the total count of inconsistent pairs, the only pairs that reveal directional positional preference

A value above 0.5 indicates first-position preference. A value below 0.5 indicates last-position preference. A value near 0.5 means that when the judge is inconsistent, the inconsistency is bidirectional and not systematically directional. This last case represents a less dangerous form of position sensitivity, closer to stochastic noise than systematic bias.

Verbosity bias is distinct from position bias but often correlated with it. LLM judges tend to score longer responses higher, even when the additional length does not convey additional information or quality. This has been observed across multiple judge models and evaluation tasks, and it creates a dangerous incentive: model developers can game LLM evaluation by making their models verbose rather than accurate or helpful.

The mechanism behind verbosity bias is not fully understood, but several factors contribute. First, length may be a proxy for effort in the judge's training data: in human-written text, longer responses often signal more thought and attention. A student who writes three pages of analysis is generally doing more work than one who writes three sentences, and human annotators who created preference data for RLHF may have absorbed this correlation. Second, longer responses tend to cover more ground, hitting topics the judge considers relevant even if some coverage is shallow or padded. If the judge uses a checklist-like internal evaluation of "did the response cover X, Y, and Z," a longer response is more likely to touch all three items even if it does so superficially. Third, formatting effects compound with length: longer responses often use more headers, bullet points, and structured presentation, which correlates with higher quality in judge training data.

Measuring verbosity bias requires controlling for quality while varying length. One approach is to create pairs of semantically equivalent responses at different lengths: take a high-quality short answer and expand it with relevant but redundant content, then ask the judge which is better. A biased judge consistently prefers the longer version. Another approach is to run a regression across many evaluation pairs. You regress judge scores on response quality (as measured by a human panel) and response length simultaneously. If length has a significant positive coefficient after controlling for quality, verbosity bias is present.

Formally, let $s_i$ be the judge score for response $i$, $q_i$ be the human-rated quality score, and $\ell_i$ be the log-length (log is used to handle the wide range of possible response lengths). Verbosity bias exists when the regression:

$$s_i = \alpha + \beta_1 q_i + \beta_2 \ell_i + \epsilon_i$$

yields $\beta_2 > 0$ with statistical significance, where:

• $s_i$: the judge score for response $i$ (the quantity we want driven only by quality)
• $q_i$: the human-annotated quality rating for response $i$ (the ground truth benchmark)
• $\ell_i$: the log-length of response $i$, capturing the length signal on a compressed scale
• $\alpha$: the intercept, representing the baseline score for a zero-quality, zero-length response
• $\beta_1$: the coefficient on quality, which should be large and positive for a well-calibrated judge
• $\beta_2$: the coefficient on length, which should be near zero for an unbiased judge; a positive value indicates verbosity bias
• $\epsilon_i$: the residual error capturing unexplained variation

The ratio $\beta_2 / \beta_1$ quantifies how much length affects scores relative to actual quality. A perfectly calibrated judge would have $\beta_2 \approx 0$, meaning length contributes nothing independent of content. Research has found $\beta_2 / \beta_1$ values as high as 0.4 in some judge models, meaning length contributes nearly as much as quality to the final verdict. A model that doubles its response length while keeping content quality constant receives nearly the same score boost as a model that substantially improves quality.

This ratio has a direct practical implication for model development. If you are fine-tuning a model and using an LLM judge with $\beta_2 / \beta_1 = 0.3$ to evaluate your checkpoints, you can artificially inflate your evaluation scores by training your model to be more verbose. This kind of evaluation gaming, sometimes called "judge hacking," is a real concern in competitive model development and one reason that LLM judge calibration audits are worth the effort.

Sycophancy is the tendency of a model to agree with or validate whatever position appears to be endorsed in the prompt. It is one of the most studied failure modes in LLM alignment research, and it shows up in evaluation as a systematic bias toward outputs that match what the judge model expects or prefers to be true. In the context of LLM-as-Judge, sycophancy manifests in several distinct ways.

The most studied form is model self-preference: a judge may favor responses generated by the same model family as itself. If GPT-4 is your judge, it may systematically overrate GPT-4-generated outputs compared to responses from other model families, because the internal representations and stylistic patterns are more familiar. The stylistic preferences baked into GPT-4 through RLHF are the same preferences it was trained to reward, so outputs that match those preferences score higher regardless of independent quality. This creates a circular evaluation loop: the judge validates outputs that resemble its own training, and model developers who use that judge receive inflated scores precisely when they replicate the judge's stylistic blind spots.

A second form is prompt-induced sycophancy: the judge's verdict is influenced by cues in the evaluation prompt that signal a preferred answer. If you include a statement like "the user found Response A more helpful" in the prompt, many judge models will shift their verdict toward Response A, even when they were explicitly instructed to ignore user preference signals and evaluate objectively. Research has demonstrated this effect in controlled conditions using what are called "authority injection" experiments, where a fake expert endorsement is added to one response before presenting it to the judge. Well-designed judges should be immune to such cues, but current models are not reliably so.

A third form is tone sycophancy: judges systematically prefer confident, authoritative-sounding responses. A response that states "the answer is X" is often rated higher than one that states "the evidence suggests X, though some uncertainty remains," even when the latter is more epistemically accurate and scientifically appropriate. This reflects a real bias in training data: human annotators, when creating preference data for RLHF, tend to reward confident responses because confidence is associated with competence in everyday language. Models absorb this pattern and apply it in evaluation contexts, penalizing appropriate uncertainty expression as if it were a sign of weakness.

The self-preference form is particularly dangerous in practice because it is invisible in any single-judge evaluation. You cannot detect it by analyzing the judge's outputs alone. You can only detect it by comparing multiple judge models from different families on the same set of examples and looking for divergences that correlate with model authorship. If all your judges come from the same model family, self-preference inflates scores uniformly across all evaluations, and you lose the ability to cross-validate or catch the inflation.

Measuring sycophancy requires careful experimental design. For self-preference measurement, you need blind evaluations where response authorship is hidden from the judge, plus comparisons across multiple judge models from different families. If Judge A consistently rates Model A's outputs higher than Judge B does, while both rate Model B's outputs similarly, that divergence provides evidence of self-preference sycophancy in Judge A. For prompt-induced sycophancy, you can inject misleading cues into evaluation prompts (such as "the user preferred Response A") and measure whether the judge updates its verdict accordingly, compared to a control evaluation with no injection. The sycophancy rate is:

$$\text{Sycophancy Rate} = \frac{\text{count of verdicts changed by spurious signal}}{N}$$

where the numerator counts how many judgments flipped toward the injected preference compared to control evaluations with no injection. Well-calibrated judges show sycophancy rates below 5 percent. Many current models exceed 15 to 25 percent in controlled experiments, meaning a substantial fraction of their verdicts can be reversed simply by telling them a user or an expert preferred the other response.

Position bias, verbosity bias, and sycophancy do not operate independently in real evaluations. They interact and compound in ways that make their combined effect more severe than any individual bias suggests.

Consider a typical evaluation scenario: Response A is placed first in the prompt, Response A is longer than Response B, and Response A is written in the same style as the judge model. All three biases now point in the same direction, amplifying each other. The judge that would give Response A a 5 percent positional advantage, a 10 percent verbosity advantage, and a 5 percent self-preference advantage is not giving it a 20 percent combined advantage. The three effects interact through the model's internal scoring mechanism in a nonlinear way. The judge may assign a 65 to 70 percent win probability to Response A based on the combination, even if Response A is objectively the lower-quality answer.

The reverse also holds. When designing experiments to measure one bias, you need to control for the others. If you measure position bias using pairs where one response happens to be systematically longer or happens to match the judge's style, your position bias estimate will be confounded. Good measurement methodology requires carefully constructing pairs where quality, length, and authorship are controlled, varying only the ordering. This is why bias measurement frameworks typically include separate datasets designed to isolate each bias and a combined dataset to measure interaction effects.