Position Bias in LLM Judges › Why Position Bias Exists

To understand why position bias occurs in LLM judges, it helps to think about what happens inside an autoregressive language model when it reads a long prompt containing two candidate answers. The model processes tokens left to right, building up a contextual representation as it goes. By the time it reaches the instruction to compare and score, the representations of Response A and Response B are not equally salient in the model's attention pattern. The attention mechanism, trained on data where position in a sequence carries real meaning, treats earlier and later content differently in ways that do not simply cancel out.

There are two primary mechanisms that contribute to position bias. The first is recency bias: models often weight recent tokens more heavily because recency is a reliable signal in natural language. In a news article, the most important conclusions often appear near the end. In an argument, the final rebuttal carries more rhetorical weight. In narrative writing, the resolution comes last. When applied to an evaluation prompt, recency manifests as a preference for whichever response appeared last. The second mechanism is primacy bias: in some contexts, the first item in a list or the first argument in a debate receives outsized weight simply because it sets the reference point for everything that follows. Human psychology exhibits a similar split depending on the task, and models trained on human-generated text absorb these contradictory patterns.

The key insight is that neither bias operates alone, and which direction dominates depends on the judge model's architecture, the nature of its training data, the length of the prompt, and even the specific comparison task. Some models show strong first-position preference. Others show strong last-position preference. Some flip direction depending on whether the evaluation prompt uses markdown headers, whether responses are separated by explicit delimiters, or whether there is a brief summarizing sentence before the final verdict. The only way to know which pattern your judge exhibits is to measure it empirically.

Position bias was first rigorously documented in the context of LLM evaluation by the MT-Bench paper (Zheng et al., 2023) and in the LMSYS Chatbot Arena analysis. Researchers found that GPT-4, when used as a judge, showed a statistically significant preference for the first response in roughly 60 to 70 percent of comparisons where it changed its verdict when the order was swapped. This was not a small or ignorable effect. It meant that the outcome of an LLM evaluation depended substantially on which team happened to submit their model's response first in the prompt template, a variable with no connection to quality whatsoever.

To build an even deeper intuition for why position bias exists, consider how a transformer's attention mechanism operates during a long-form evaluation prompt. When the model reads a prompt containing 800 tokens of context and reaches the final instruction "Which response is better? Answer A or B," the attention patterns over that context are not uniform. The model attends more strongly to some tokens than others, and both position within the sequence and proximity to the current generation step influence those weights.

Research on attention patterns in long contexts has shown that many transformer models exhibit a characteristic attention shape sometimes called the "U-curve": tokens at the very beginning and very end of a long sequence receive disproportionately high attention, while tokens in the middle tend to be underweighted. This is sometimes called the "lost in the middle" phenomenon. For an evaluation prompt where Response A occupies the beginning and Response B occupies the middle-to-end, Response A benefits from primacy attention and Response B benefits from recency attention, but the magnitudes are not symmetric across model families and sizes.

Instruction-tuned models reduce this asymmetry but do not eliminate it. When a model is fine-tuned on preference data where human annotators reviewed pairs in a fixed order, the model learns associations between positional patterns and quality assessments that were never intended to be part of the evaluation rubric. Those learned associations surface as position bias.

Position bias is not purely a product of model architecture. The structure of the evaluation prompt significantly shapes how much bias appears. Several design choices systematically amplify or dampen position effects:

Asymmetric framing occurs when the prompt refers to the two responses using unequal language, such as "here is the first response" versus "here is another response." The word "first" subtly signals that the ordering is meaningful, which may cause the model to treat position as a legitimate evaluation signal.

Distance between responses affects how coherently the model can compare them. When Response A and Response B are separated by a long rubric section or detailed evaluation criteria, the model's contextual representation of A has partially faded by the time it fully processes B. Shorter, more structured prompts with explicit separators reduce this effect.

The label convention matters more than it might seem. Using labels like "Response A" and "Response B" is better than "Response 1" and "Response 2" because ordinal numbers ("first," "second") are more strongly associated with ranked sequences in training data. Using neutral labels, or randomly assigning labels like arbitrary colors or codes, further weakens the positional signal.

The position of evaluation criteria affects which response the model mentally compares to the rubric most vividly. If evaluation criteria are stated after both responses, the most recent response (B) is compared more directly to the freshly active criteria. If criteria come before both responses, both are evaluated against the same contextually distant rubric.

Understanding these design factors is practically useful: careful prompt engineering can reduce position bias even without any algorithmic mitigation. But it cannot eliminate it entirely, which is why measurement and swapping remain necessary.