Test failure masks following failures

[c1728p9]

One failing test can make following tests appear to pass. An example of this can be found here:
c1728p9/mbed-os@3c38cc6

@PrzemekWirkus, @bridadan

[PrzemekWirkus]

@adbridge Anna, please have a look when you are back from training.

[Mirrored to Jira]

[adbridge]

@c1728p9 Please provide more detailed summary of problem. In your example I just see 2 asserts in succession (which can't every pass anyway as discussed elsewhere) so what do you mean by 2 tests? Do you mean that the first assert in your example fails but the second then passes ? Please provide a link to the full test being run, which compiler and target was used and what platform you ran on as a bare minimum as part of the actual issue raised.

[Mirrored to Jira]

[adbridge]

Aside from the above comment , having looked at the full source file for your supplied example you have a custom failure handler which is run at the end of each of your test cases if they fail. This handler has the following code

utest::v1::status_t greentea_failure_handler(const Case *const source, const failure_t reason) {
    greentea_case_failure_abort_handler(source, reason);
    return STATUS_CONTINUE;
}

By returning STATUS_CONTINUE you are instructing the test framework to run the next test case even if the previous one failed. So you should get a test failure and then the rest still run....

[Mirrored to Jira]

[c1728p9]

Try running the c string test on a K64F and you should see the problem. Only one test fails and all subsequent tests within the file pass.

[Mirrored to Jira]

[adbridge]

Are the subsequent tests supposed to fail ?

[Mirrored to Jira]

[c1728p9]

Yeah, I added the "TEST_ASSERT(0);" which should always fail, but in this case it passes.

[Mirrored to Jira]

[adbridge]

Ah ok now I understand the problem, will investigate.

[Mirrored to Jira]

[c1728p9]

Sorry for any confusion. Thanks for looking into it 😃

[Mirrored to Jira]