Position Bias in LLM Judges › Summary

Position bias, verbosity bias, and sycophancy are not edge cases in LLM evaluation. They are systematic properties of current judge models that affect every evaluation pipeline at some level, and they require active measurement and mitigation rather than passive acceptance.

The key takeaways from this chapter are:

• Position bias causes LLM judges to prefer responses in particular ordinal positions, independent of quality. Swap consistency is the primary diagnostic metric. Answer swapping is the standard mitigation, though it doubles evaluation cost and converts position-sensitive comparisons to ties.
• Verbosity bias causes judges to prefer longer responses, creating incentives for padding over precision. It can be measured via quality-controlled regression and partially mitigated by length normalization or post-hoc regression adjustment.
• Sycophancy causes judges to agree with perceived consensus, including stylistic self-agreement with outputs from the same model family. Multi-judge consensus across different model families and cross-family comparisons reduce but do not eliminate this bias.
• Mitigation is partial, not complete. Swapping doubles evaluation cost, verbosity normalization distorts legitimate length signals, and no current mitigation fully removes sycophancy without basic training changes.
• Biases compound. A judge exhibiting all three biases simultaneously is substantially less reliable than the individual bias rates suggest, because the three effects can reinforce each other when they happen to point in the same direction for a given comparison.
• LLM judges are best used within systems, not as standalone arbiters. Calibration against human annotations, multi-judge consensus, per-dimension scoring, and adversarial bias probes together make automated evaluation reliable enough for production use.