MBPP › Code Evaluation Challenges

Execution-based evaluation introduces complexities that purely textual benchmarks avoid. When we run generated code, we enter the domain of software engineering with all its attendant concerns about security, determinism, and resource management. Unlike evaluating text generation, where the worst-case scenario is creating nonsensical output, executing arbitrary code can actively harm the evaluation environment, consume excessive resources, or introduce security vulnerabilities. These concerns are not hypothetical: models trained on large code corpora have occasionally generated code that performs file system operations, initiates network connections, or triggers infinite recursion.

Executing arbitrary code generated by language models requires reliable isolation. Models might produce code that accesses the file system inappropriately, attempts network connections to exfiltrate data, consumes excessive memory through large data structure creation, contains infinite loops that never terminate, or exploits Python interpreter vulnerabilities to escape the sandbox.

MBPP evaluation requires sandboxed execution environments. Production-grade evaluation systems typically combine multiple layers of defense:

• Process isolation: Running untrusted code in a separate process that can be killed if it exceeds time limits. This gives basic protection against infinite loops and runaway computation.
• Containerization: Running the subprocess inside a Docker container with restricted capabilities, no network access, and a read-only file system except for a designated temporary directory. This prevents file system manipulation and network access.
• Resource limits: Enforcing memory caps (typically 256 MB to 1 GB), CPU time limits, and strict timeouts (typically 2 to 5 seconds per execution). These prevent denial-of-service scenarios where a single problem consumes all available resources.
• Restricted Python: Using environments like RestrictedPython that disable dangerous builtins, or running code in a custom Python interpreter that blocks imports of modules like os, subprocess, and socket.
• Virtual machine isolation: For the most stringent security requirements, running entire VMs per evaluation batch. This keeps even container escapes cannot affect the host system.

The sanitized subset of MBPP is particularly useful because it reduces (though does not eliminate) the risk of harmful patterns in the ground truth solutions. Even with sanitized reference solutions, generated model outputs remain potentially dangerous, requiring strict isolation between the evaluation harness and the execution environment.

Python code can behave non-deterministically due to random number generation, dictionary ordering in certain contexts, timing-dependent operations, and mutable global state. Non-determinism creates problems for reproducibility in research, since the same code might pass tests on one run but fail on another.

MBPP mitigates this by focusing on pure functions with deterministic outputs for given inputs. However, evaluating models that use randomization requires careful test design. Tests must either set random seeds explicitly before calling the function, allow for multiple valid outputs by testing properties rather than exact equality, or verify that the output belongs to a set of acceptable values.

Non-determinism presents a particular challenge for pass@k estimation. If a function's test result varies between runs, the estimate of $c$ (the number of correct samples) becomes unstable, introducing noise into the metric. The standard practice is to run each candidate exactly once against the test suite, accepting that any non-determinism in the test outcomes adds small amounts of noise to the final metric. For most MBPP problems, this is not a concern since the problems are defined to have deterministic correct answers.

One of the most contentious issues in code evaluation is how to handle partially correct solutions. A generated function might pass basic cases but fail edge cases, handle the main logic but miss error handling, return correct results but with incorrect types, implement the right algorithm with an off-by-one error, or solve a subtly different problem than the one specified.

MBPP, like HumanEval, uses a binary pass/fail criterion: either all assertions pass, or the solution is considered incorrect. This simplifies evaluation but loses granularity. A model that generates almost-correct code, failing only on one edge case out of ten tests, receives the same score as one that produces complete nonsense.

Alternative approaches have been proposed. Fractional scoring awards partial credit based on the percentage of test cases passed, so a function that passes 9 out of 10 tests receives a score of 0.9 instead of 0. Severity-based evaluation distinguishes between failure types: syntax errors (the code does not even parse), runtime errors (the code crashes), and logic errors (the code runs but produces wrong answers). Each type shows different levels of model capability. Property-based testing, rather than testing specific input-output pairs, verifies that the generated code satisfies high-level properties, such as "the output is always sorted" or "the output contains exactly the same elements as the input."

These alternatives add complexity and sometimes subjectivity. The binary approach remains standard because it correlates well with the practical utility of code: in production systems, a function that fails 10% of cases is often as unusable as one that fails 100% when those failures occur on real inputs. While a human developer might debug a nearly-correct solution more easily than starting from scratch, the benchmark setting does not model that iterative repair process.

The quality of MBPP evaluation depends entirely on the quality of the test cases. Crowd-sourced test cases vary materially in their coverage of the problem specification. Some problems have complete suites checking edge cases; others verify only the most basic functionality.

This variability affects model rankings in subtle ways. A model might reach a high MBPP score by generating "obvious" solutions that satisfy basic tests while failing edge cases that more thorough testing would catch. The sanitized subset partially addresses this by including only problems where reviewers confirmed that the test cases adequately cover the specification, but even expert-verified tests cannot prove absence of bugs for all possible inputs.

The coverage problem has motivated augmented versions of MBPP. The MBPP+ dataset from the EvalPlus project augmented the original test cases by generating additional inputs using a combination of LLM-generated tests and mutation testing, then having human experts verify them. When evaluated on MBPP+, models score lower than on the original MBPP, suggesting that many apparent successes on the original benchmark reflect weak test coverage rather than real code correctness. This gap between MBPP and MBPP+ performance is an important calibration tool: it quantifies how much a model's apparent capability depends on the leniency of the evaluation.