In the TestEval paper, three criteria are mentioned for test case correctness evaluation:
Syntactic correctness
Execution correctness
Assertion correctness
The current implementation in eval_overall.py properly evaluates:
Syntactic correctness (using compile(testcase,'<string>','exec'))
Execution correctness (using the execute() function, which returns True for both normal execution and assertion errors)
However, the assertion correctness metric mentioned in the paper isn't implemented.
In the TestEval paper, three criteria are mentioned for test case correctness evaluation:
Syntactic correctness
Execution correctness
Assertion correctness
The current implementation in
eval_overall.pyproperly evaluates:Syntactic correctness (using
compile(testcase,'<string>','exec'))Execution correctness (using the
execute()function, which returns True for both normal execution and assertion errors)However, the assertion correctness metric mentioned in the paper isn't implemented.