HumanEval › The Evolution Toward HumanEval+

The original HumanEval test cases were designed to be correct but not exhaustive. A natural question arose: how many solutions that pass HumanEval's tests are wrong on a more complete suite? In 2023, researchers at the University of Illinois developed HumanEval+, which augmented each HumanEval problem with up to 80 times more test cases generated using the EvoEval mutation-based approach. The results were sobering.

Across a range of state-of-the-art models, HumanEval+ scores consistently fell below the original HumanEval scores by a real margin. Models that appeared to solve 80% of problems correctly on the original benchmark turned out to solve only 70-75% correctly on the expanded test suite. This gap varied by model: some models exhibited highly reliable solutions that passed the expanded tests, while others relied on solutions that happened to satisfy the sparse original tests but failed on edge cases that were not directly tested.

This finding has two practical consequences. First, HumanEval scores from 2021 and 2022 should be interpreted with some skepticism: part of the apparent progress may reflect models learning patterns that satisfy sparse tests rather than generating correct solutions across edge cases. Second, for evaluation purposes, HumanEval+ is a more rigorous standard, and recent papers increasingly report results on both benchmarks.

The test augmentation methodology used in HumanEval+ also introduced a new evaluation paradigm: instead of hand-crafting every test case, generate them programmatically using mutation testing or property-based testing techniques. A mutation involves making small, systematic changes to the canonical solution (swapping < for <=, negating a condition, changing an off-by-one index) and checking whether the model's solution correctly handles the mutated inputs. This automated approach to test generation scales more cheaply than human curation and can cover edge cases that human test writers might overlook.