MBPP › Code Implementation: Loading and Evaluating MBPP
Language AI HandbookAll chapters MBPP › Code Implementation: Loading and Evaluating MBPPOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We implement a complete pipeline for loading the MBPP dataset and running evaluation. We will use the HuggingFace datasets library, which hosts the sanitized MBPP split, and build a minimal evaluation harness that shows the key engineering patterns.
# Load MBPP dataset from HuggingFace
try:
from datasets import load_dataset
raise ImportError("Skipping download for demonstration")
except ImportError:
print("Using mock data for demonstration")
mbpp = [
{
"task_id": 1,
"text": "Write a function to add two numbers.",
"code": "def add_numbers(a, b):\n return a + b",
"test_list": [
"assert add_numbers(2, 3) == 5",
"assert add_numbers(-1, 1) == 0",
],
},
{
"task_id": 2,
"text": "Write a function to find the maximum value in a list.",
"code": "def find_max(lst):\n return max(lst)",
"test_list": [
"assert find_max([1, 3, 2]) == 3",
"assert find_max([-1, -5, -2]) == -1",
],
},
{
"task_id": 3,
"text": "Write a function to check if a string is a palindrome.",
"code": "def is_palindrome(s):\n return s == s[::-1]",
"test_list": [
"assert is_palindrome('racecar') == True",
"assert is_palindrome('hello') == False",
],
},
]Dataset size: 3 problems Example problem (task_id=1): Text: Write a function to add two numbers. Tests: ['assert add_numbers(2, 3) == 5', 'assert add_numbers(-1, 1) == 0']
Safe Execution Environment
To safely evaluate generated code, we need a sandboxed execution environment. The following implementation shows the subprocess-based approach, which gives process-level isolation:
def execute_code_safely(
code: str, test_cases: List[str], timeout: int = 5
) -> Tuple[bool, str]:
"""
Execute Python code with test cases in a subprocess for basic isolation.
Combines the function code with assertion-based test cases,
writes to a temporary file, and runs in a separate process with
a time limit. Returns success status and any error message.
Args:
code: The Python code to execute (typically function definitions)
test_cases: List of assertion statements to verify correctness
timeout: Maximum execution time in seconds
Returns:
Tuple of (success: bool, error_message: str)
"""
full_script = code + "\n\n" + "\n".join(test_cases)
with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
f.write(full_script)
temp_path = f.name
try:
result = subprocess.run(
["python3", temp_path],
capture_output=True,
text=True,
timeout=timeout,
env={"PYTHONPATH": "", "PATH": os.environ.get("PATH", "")},
)
os.unlink(temp_path)
success = result.returncode == 0
error_msg = result.stderr.strip() if not success else ""
return success, error_msg
except subprocess.TimeoutExpired:
os.unlink(temp_path)
return False, "Timeout: execution exceeded time limit"
except Exception as e:
if os.path.exists(temp_path):
os.unlink(temp_path)
return False, f"Execution error: {str(e)}"This execution wrapper gives basic isolation through subprocess separation: the untrusted code runs in a child process that can be killed on timeout, and any exceptions from the child process are captured and reported rather than crashing the evaluation harness. The env parameter restricts the environment variables visible to the subprocess. For research and development purposes, this level of isolation is often sufficient. Production systems handling untrusted model outputs would add Docker containerization as an additional layer.
# Verify the execution wrapper works on a known-correct solution
test_code = mbpp[0]["code"]
test_assertions = mbpp[0]["test_list"]
success, error = execute_code_safely(test_code, test_assertions)Execution result for 'Write a function to add two numbers.': PASS
Computing pass@k Across a Dataset
Now let us implement the full evaluation pipeline that generates multiple candidates and computes pass@k. In a real evaluation, the generation step would call a language model API; here we show the evaluation logic with mock candidates.
def evaluate_problem_with_mock_candidates(
problem: Dict[str, Any],
correct_rate: float = 0.3,
n_samples: int = 10,
seed: int = 42,
) -> Dict[str, Any]:
"""
Evaluate a problem by testing mock candidates with a given correct rate.
In production, replace the mock generation with a real model call.
Args:
problem: MBPP problem dict with 'task_id', 'code', 'test_list'
correct_rate: Fraction of candidates that will be correct (mock)
n_samples: Number of candidate solutions to generate
seed: Random seed for reproducibility
Returns:
Dictionary with task_id, correct_samples, total_samples, pass_at_k
"""
import random
rng = random.Random(seed)
test_cases = problem["test_list"]
reference_code = problem["code"]
correct_count = 0
for _ in range(n_samples):
# Mock generation: either correct reference code or a broken version
if rng.random() < correct_rate:
candidate = reference_code
else:
# Broken version: correct signature but always returns None
first_line = reference_code.strip().split("\n")[0]
candidate = first_line + "\n return None"
is_correct, _ = execute_code_safely(candidate, test_cases)
if is_correct:
correct_count += 1
n = n_samples
c = correct_count
pass_at_k = {
f"pass@{k}": calculate_pass_at_k(n, c, k) for k in [1, 5, 10] if k <= n
}
return {
"task_id": problem["task_id"],
"correct_samples": c,
"total_samples": n,
"pass_at_k": pass_at_k,
}Problem 1: 4/10 correct pass@1: 0.400 pass@5: 0.976 pass@10: 1.000
Problem 2: 3/10 correct pass@1: 0.300 pass@5: 0.917 pass@10: 1.000
Problem 3: 6/10 correct pass@1: 0.600 pass@5: 1.000 pass@10: 1.000
The separation between generation and evaluation is a key design principle. It allows you to swap in different models, prompting strategies, or decoding configurations while maintaining identical evaluation methodology. This separation also makes it easy to add result caching, distributed evaluation across multiple machines, or streaming evaluation that reports results as they become available rather than waiting for all samples to complete.
Aggregating Results Across Problems
After evaluating all problems, we aggregate the per-problem metrics into dataset-level pass@k scores. The standard approach is to average pass@k across all problems:
def aggregate_results(problem_results: List[Dict]) -> Dict[str, float]:
"""
Compute dataset-level pass@k by averaging across problems.
Args:
problem_results: List of per-problem evaluation dictionaries
Returns:
Dictionary mapping metric names to average scores
"""
if not problem_results:
return {}
all_metrics = set()
for result in problem_results:
all_metrics.update(result["pass_at_k"].keys())
aggregated = {}
for metric in sorted(all_metrics):
scores = [
r["pass_at_k"][metric]
for r in problem_results
if metric in r["pass_at_k"]
]
if scores:
aggregated[metric] = sum(scores) / len(scores)
return aggregatedDataset-level metrics (mock evaluation with 40% correct rate): pass@1: 0.433 pass@10: 1.000 pass@5: 0.964
In practice, researchers generate samples per problem and report the resulting pass@k for . The pass@1 score is the most directly interpretable: it is the probability that the model's first output is correct when evaluated on a random MBPP problem. For state-of-the-art models in 2024, pass@1 on the MBPP sanitized set reaches above 80%, a remarkable improvement from the 50-60% range achieved by early GPT-3-scale models.