MBPP › Dataset Construction and Philosophy

MBPP emerged from a recognition that expert-written benchmarks, while high-quality, might not capture the full distribution of programming tasks that practitioners encounter daily. The researchers crowdsourced programming problems from a pool of Python developers at varying skill levels, creating a dataset that shows how everyday programmers describe and solve tasks. The collection process asked contributors to write programming problems they had personally encountered or imagined, then give reference solutions and test cases. This bottom-up approach produced a dataset with a qualitatively different character than HumanEval.

The philosophical move from expert curation to crowd-sourcing is a bet on ecological validity. By collecting problems the way they naturally occur in development environments, MBPP aims to measure whether models can handle the irregular, often ambiguous specifications that working developers give. This diversity comes with trade-offs. While expert benchmarks ensure consistent quality and complete edge-case coverage, crowd-sourced data introduces variability in problem difficulty, description clarity, and test thoroughness. MBPP embraces this variability as a feature rather than a bug, arguing that reliable code generation requires handling imperfect specifications and incomplete requirements.

Consider the perspective of a developer using a code generation tool. They are rarely in the mindset of constructing a carefully defined algorithmic problem. They need help with a specific, immediate task: "write a function to strip HTML tags from this string" or "create a function that groups these objects by date." These informal, task-oriented descriptions are exactly what MBPP captures. A model that performs well on MBPP has learned to interpret human intent even when the specification leaves details unstated, infer reasonable design choices when the prompt is ambiguous, and generate code that solves the practical problem rather than merely satisfying formal constraints.

The full MBPP dataset contains 974 programming problems divided into subsets based on quality and verification levels. Understanding these splits matters for interpreting benchmark results, as different papers report scores on different subsets and direct comparisons require awareness of which split was used.

The three primary divisions are:

• Sanitized subset (427 problems): Problems that human reviewers manually verified to have correct solutions and reasonably complete test cases. This is the most commonly reported split for benchmarking newer models. The smaller size shows the labor-intensive verification process.
• Crowd-sourced subset (547 problems): Problems as originally submitted, without manual verification. These may contain ambiguous specifications, subtly incorrect reference solutions, or test cases that do not fully capture the intended functionality.
• Augmented versions: Later releases, such as MBPP+ from the EvalPlus project, include additional test cases generated to improve coverage and reduce the fraction of "false positive" evaluations where models pass insufficient tests.

Reporting practices in the literature vary. Some papers report pass@1 on the full 974-problem set, while others use only the 427-problem sanitized split. When comparing scores across papers, you should always verify which split was used, as scores on the sanitized subset tend to be slightly lower due to more rigorous test cases.

Each problem in MBPP follows a consistent structure that makes automated evaluation straightforward:

• Task ID: A unique integer identifier for the problem, letting reproducible reporting of per-problem results.
• Text: The natural language description of the programming task, serving as the prompt given to the model. This is typically one to three sentences in a conversational, informal style.
• Code: A reference solution implementing the described functionality. This is ground truth but is not shown to the model during evaluation.
• Test Cases: A list of Python assertion statements that verify the correctness of any proposed solution. These are the executable unit tests that determine pass or fail.
• Test Setup: Any necessary imports or initialization code that must run before the test assertions. This allows the test cases to import the function being tested.

Keeping the prompt (what the model sees), the reference solution (what a correct answer looks like), and the test cases (how correctness is verified) separate makes MBPP easy to integrate into evaluation pipelines. The test cases are particularly important: they are the actual mechanism of evaluation, not the reference solution. A model's generated code is run against the test assertions, and the reference solution is used only for comparison purposes, not for grading.

Unlike HumanEval, which focuses on algorithmic interview-style problems, MBPP spans a broader range of programming tasks with generally lower algorithmic complexity. The "Mostly Basic" moniker is not false modesty: the dataset stresses basic programming competencies rather than competitive programming puzzles. This focus aligns with the reality that professional programming involves substantial amounts of routine data processing, format conversion, and utility function implementation alongside occasional complex algorithmic work.

The problems in MBPP exhibit several characteristics that distinguish them from other benchmarks:

• Conversational descriptions: Problems are often phrased informally, resembling how you might ask a programming assistant for help rather than how you would write a formal specification.
• Standard library focus: Solutions typically rely on Python's standard library, particularly string methods, list operations, the datetime module, and basic data structure operations. This keeps the problems self-contained without requiring knowledge of specialized packages.
• Single-function scope: Most problems ask for a single function implementation, though some require brief scripts. This narrowly defined scope makes evaluation clear but may underrepresent real-world tasks that span multiple functions.
• Self-contained nature: Each problem includes all necessary context without external dependencies, making batch evaluation straightforward.
• Breadth over depth: MBPP problems rarely require deep algorithmic insight, but they cover a wide range of Python programming scenarios, from string manipulation and list sorting to simple number theory and basic data structure operations.

The typical MBPP problem asks for something like: "Write a function to find the second largest element in a list," "Write a function to check if a string is a palindrome ignoring whitespace," or "Write a function to convert a decimal number to binary representation." These are the kinds of tasks that appear in tutorials, coding exercises for beginners, and everyday utility scripting. They are not trivial, but they do not demand the advanced reasoning that HumanEval's algorithmic problems require.

This scope makes MBPP particularly useful for evaluating instruction-tuned models designed to serve as coding assistants. When you ask an AI to "write a function to convert this CSV format," you are not giving formal specifications or function signatures. You are giving conversational, often incomplete descriptions. MBPP tests a model's ability to bridge the gap between informal human intent and executable machine instructions, a capability that matters more for deployed systems than the ability to solve algorithmic puzzles under pressure.

The MBPP problems cluster into several informal categories that reveal the breadth of everyday Python programming. While the dataset does not come with official category labels, researchers have analyzed the distribution and identified recurring patterns. The major categories include the following types.

String operations and text processing constitute a large portion of the dataset. Problems ask for functions to reverse words, count vowels, extract substrings matching certain patterns, format text according to rules, and manipulate strings in various ways. These problems test whether models understand Python's string API and can combine multiple operations to reach a desired transformation.

Mathematical and numerical functions form another substantial category. Problems involve computing properties of numbers (checking primality, finding factors, computing digit sums), performing geometric calculations (area, perimeter, distance), and implementing simple numerical algorithms (binary search, basic sorting variants). These problems require translating mathematical descriptions into executable code without necessarily requiring deep algorithmic insight.

List and collection operations appear frequently, covering sorting with custom keys, filtering based on predicates, aggregating values, finding extremes with various conditions, and transforming between different collection types. Python's rich built-in collection operations make many of these problems amenable to concise solutions, testing whether models know the standard library well.

Date and time operations stand for a practically important category that is often surprisingly tricky. Converting between date formats, computing differences between dates, finding the day of week for a given date, and performing calendar arithmetic require understanding Python's datetime module and its interface.

Basic algorithmic problems round out the set, including simple dynamic programming tasks, graph traversals on small inputs, and recursive computations. These overlap with HumanEval in spirit but tend to be simpler in scope and described in less formal terms.