Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

MBPP › Visualizing Evaluation Dynamics

Michael BrenndoerferMarch 3, 20262 min read

Language AI HandbookAll chapters

Part 54 of 67Benchmark Evaluation

Chapter 419MBPP › Visualizing Evaluation Dynamics

Reading section 9 of 12 · Visualizing Evaluation Dynamics · 2 minSection 9 of 12 · 2 min

Section 9 of 12 · 2 min read

MBPP › Visualizing Evaluation DynamicsOne section · Sections in this chapter9 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Beyond the single pass@k number, understanding how evaluation metrics behave across different settings helps calibrate expectations and design better experiments.

Out[23]:

Visualization

Contour heatmap of pass@k probability as a function of k and base success rate. — Contour plot showing pass@k probability across a range of k values (x-axis) and base success rates c/n (y-axis). Darker blue regions indicate higher pass@k. Even models with low single-sample success rates (5-15%) can reach high pass@k when k is large, which motivates inference-time scaling strategies that generate many candidates and select among them.

The heatmap reveals an important structural property of pass@k: even models with low base success rates (5-10% per sample) can reach high pass@k when $k$ is large. This is the rationale behind inference-time scaling approaches, where generating many candidates and selecting among them using a verifier or test-based filtering can sharply improve effective performance. The contour lines also reveal that the returns to increasing $k$ are diminishing: going from $k=1$ to $k=10$ produces a much larger improvement than going from $k=40$ to $k=50$ , especially for high base success rates.