MBPP › MBPP: Mostly Basic Python Programming

When we evaluate whether a language model can write code, we need benchmarks that reflect real programming tasks. While HumanEval gives us carefully hand-crafted problems written by experts, it covers only a narrow range of programming tasks. We need larger, more diverse collections to understand how models generalize across different problem types, difficulty levels, and coding styles. Expert-crafted benchmarks reveal one dimension of capability; crowd-sourced benchmarks reveal another. Each illuminates failure modes the other obscures.

MBPP (Mostly Basic Python Programming) fills this gap. Created by researchers at Google and published in 2021, MBPP contains nearly 1,000 Python programming problems sourced from a crowd of developers rather than curated by a small group of experts. This move from expert craftsmanship to crowd-sourced diversity reveals different failure modes in code-generating models and gives a complementary view of programming capabilities. Where expert-written benchmarks might reflect idealized or algorithmically advanced scenarios, crowd-sourced data captures the messy, practical reality of day-to-day software development: string manipulations, date arithmetic, file parsing, and data structure transformations that comprise the bulk of what working programmers write.

The creation of MBPP came at a point where practice changed. By 2021, language models had shown impressive ability on HumanEval's structured algorithmic problems, yet practitioners remained skeptical about whether these capabilities would transfer to real programming assistance. If a developer types a comment describing a function they want and presses a completion shortcut, the model receives an informal, conversational specification, not a carefully structured docstring with type hints. MBPP was designed precisely to test this more realistic scenario, bridging the gap between laboratory benchmarks and deployed coding assistants.

In this chapter, we will explore the structure of the MBPP dataset, examine how it differs from HumanEval in both construction and evaluation philosophy, work through the mathematics of pass@k evaluation in detail, and confront the unique challenges of execution-based code evaluation. Understanding these distinctions matters because the choice of benchmark fundamentally shapes what we optimize for when training code-generating models. If we train exclusively on expert-curated algorithmic puzzles, we might produce models that excel at interview questions but struggle with simple utility scripts. Conversely, overfitting to basic crowd-sourced tasks might leave models unprepared for complex architectural decisions. The full picture of programming competence requires both kinds of evaluation.