MBPP › Worked Example: An MBPP Problem End to End
Language AI HandbookAll chapters MBPP › Worked Example: An MBPP Problem End to EndOne section · Sections in this chapter7 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We examine a typical MBPP problem to understand the complete evaluation flow. Consider a problem asking for a function to calculate the number of days between two dates. This problem is representative of MBPP's practical, utility-function style: it asks for something you might need in a data processing script, uses a standard library module, and has a clear correct answer that is easy to verify.
# Example MBPP-style problem structure
mbpp_example = {
"task_id": 123,
"text": "Write a function to calculate the number of days between two dates. The function should accept two date strings in the format 'YYYY-MM-DD' and return the absolute number of days between them.",
"code": """
from datetime import datetime
def days_between(date1, date2):
d1 = datetime.strptime(date1, '%Y-%m-%d')
d2 = datetime.strptime(date2, '%Y-%m-%d')
return abs((d2 - d1).days)
""",
"test_list": [
"assert days_between('2023-01-01', '2023-01-10') == 9",
"assert days_between('2023-01-10', '2023-01-01') == 9",
"assert days_between('2020-01-01', '2020-01-02') == 1",
],
"test_setup_code": "from datetime import datetime",
}Notice several things about this problem structure. The prompt is informal and conversational. The reference solution is straightforward but uses a specific approach (string parsing with strptime). The tests verify the core functionality and test for commutativity (both orderings return the same result), but they do not test edge cases like invalid date strings, dates that span years, or very large date differences. This limited test coverage is characteristic of many crowd-sourced MBPP problems.
When evaluating a model on this problem, we would follow a four-step process. First, generate candidate solutions by sampling the model multiple times given the text prompt. Second, execute each candidate with the test assertions in an isolated environment. Third, check whether all assertions pass without exception. Fourth, compute pass@k based on how many candidates succeeded.
def evaluate_candidate(candidate_code, test_setup, test_list):
"""
Execute candidate code against test cases.
Returns True if all tests pass, False otherwise.
Note: Uses exec for illustration. Production code should use
subprocess-based isolation for security.
"""
tests_code = "\n".join(test_list)
full_code = f"""
{test_setup}
{candidate_code}
{tests_code}
"""
try:
namespace = {}
exec(full_code, namespace)
return True
except Exception:
return False
# Simulate evaluation with two candidate solutions
candidate_correct = """
def days_between(date1, date2):
from datetime import datetime
d1 = datetime.strptime(date1, '%Y-%m-%d')
d2 = datetime.strptime(date2, '%Y-%m-%d')
return abs((d2 - d1).days)
"""
candidate_incorrect = """
def days_between(date1, date2):
# Bug: only compares day numbers within the month, ignores month and year
return abs(int(date2.split('-')[2]) - int(date1.split('-')[2]))
"""
results = {
"correct_implementation": evaluate_candidate(
candidate_correct,
mbpp_example["test_setup_code"],
mbpp_example["test_list"],
),
"buggy_implementation": evaluate_candidate(
candidate_incorrect,
mbpp_example["test_setup_code"],
mbpp_example["test_list"],
),
}correct_implementation: PASS buggy_implementation: PASS
The correct implementation uses strptime to parse date strings and computes the difference as a timedelta object, then takes the absolute value of its .days attribute. The buggy implementation only compares day numbers within the month.
Interestingly, the buggy solution passes all three tests in this example, because the tests only check cases within the same month where day subtraction happens to give the right answer. A case like days_between('2023-01-01', '2023-02-01') would expose the bug (returning 0 instead of 31), but that case is not in the test suite. This illustrates the coverage problem concretely: weak test suites allow incorrect solutions to appear correct, inflating evaluation scores.