Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Position Bias in LLM Judges › Visualizations

Michael BrenndoerferMarch 11, 20266 min read

Language AI HandbookAll chapters

Part 55 of 67Human and Model Evaluation

Chapter 427Position Bias in LLM Judges › Visualizations

Reading section 6 of 9 · Visualizations · 6 minSection 6 of 9 · 6 min

Section 6 of 9 · 6 min read

Position Bias in LLM Judges › VisualizationsOne section · Sections in this chapter6 / 9 ↓

Sections in this chapter9

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The four visualizations below illuminate the bias patterns we have measured: how position and verbosity bias interact to reduce swap consistency, how bias degrades judge accuracy, how verbosity bias distorts individual comparisons, and how mitigation effectiveness changes as bias strength grows.

First, a heatmap showing swap consistency across different combinations of position bias and verbosity bias:

Out[13]:

Visualization

Heatmap of swap consistency rates across position and verbosity bias strengths, with values declining toward bottom-right. — Swap consistency rates across combinations of position bias strength and verbosity bias strength. Each cell shows the fraction of comparisons that produce the same verdict regardless of response ordering. Darker red cells indicate lower consistency, meaning more decisions flip when order is reversed. Both biases independently reduce consistency, and their combination produces the lowest values in the bottom-right region.

Second, a bar chart comparing accuracy and position preference rate across judge configurations, showing how each bias degrades reliability:

Out[14]:

Visualization

Bar chart comparing accuracy and position preference rate for three judge configurations, showing how bias degrades accuracy and skews position preferences. — Comparison of judge accuracy and position-A preference rate across three bias configurations: no bias, position bias only, and combined position plus verbosity bias. The dashed line marks the expected position preference rate of 0.50 for an unbiased judge. Accuracy falls as bias grows, and position-A preference climbs above the neutral baseline, confirming systematic first-position preference.

Third, a scatter plot showing the relationship between response length difference and quality difference, colored by judge verdict. Points in the quadrant where one response is longer but the other is higher quality reveal verbosity bias directly:

Out[15]:

Visualization

Scatter plot showing response length difference versus quality difference, colored by judge verdict to reveal verbosity bias patterns. — Scatter plot of log-length difference versus quality difference between response pairs, colored by judge verdict. Blue points show comparisons where the judge picked response A, red points where it picked B. Points in the upper-left quadrant (A higher quality, B longer) colored red indicate verbosity bias: the judge chose the longer but lower-quality response. The density of red points in the upper-left reveals how severely verbosity inflates verdicts.

Fourth, a line chart showing how swap consistency and accuracy evolve as position bias strength increases, comparing unmitigated evaluation with swap-mitigated evaluation:

Out[16]:

Visualization

Line chart comparing swap consistency, unmitigated accuracy, mitigated accuracy, and tie rate across increasing position bias strengths. — Swap consistency and judge accuracy as position bias strength increases from 0 to 0.5, comparing single-ordering (unmitigated) evaluation with swap-mitigated evaluation. Mitigated accuracy stays higher than unmitigated accuracy across all bias levels, but the tie rate (dotted line) grows steadily, which reflects the increasing proportion of comparisons that require human resolution. At high bias levels, more than one-third of comparisons become ties.