Skip to content

Conversation

@jonpryor
Copy link
Contributor

@jonpryor jonpryor commented Mar 9, 2017

While reviewing PR 445 I noticed that it said that 41 tests had
been run, and 0 failed. Compare to e.g. PR 474, which has 164
passing tests.

Something isn't right here.

Further investigation showed that for PR 445, execution on the
emulator had crashed ("Segmentation fault"). The error was ignored.

Ignoring errors was in fact By Design™; see commit 6358a64.

So what we need is for the RunUnitTestApks target to not report
errors, while still reporting errors.

Solve this conundrum by updating the <RunInstrumentationTests/> task
so that when the nunit2-results-path value can't be found, it
reports a successful run, and sets the
RunInstrumentationTests.FailedToRun property to the name of the
component that failed. This allows the ReleaseAndroidTarget target
to check for any failures, and report an <Error/> after shutting
down the emulator instance.

This should allow the build to "fail" when nunit2-results-path can't
be found -- a sign that the .apk unit tests didn't finish executing
-- while also allowing the emulator to be shut down.

Aside: xbuild doesn't support ContinueOnError="ErrorAndContinue",
nor does it set <Output/> properties whenever a task fails.
Consequently, the <RunInstrumentationTests/> task must return true
when nunit2-results-path isn't found, because there's no other way
to convince xbuild to collect the <Output/> values.

While reviewing [PR 445][0] I noticed that it said that *41* tests had
been run, and 0 failed. Compare to e.g. [PR 474][1], which has *164*
passing tests.

Something isn't right here.

Further investigation showed that for PR 445, execution on the
emulator had crashed ("Segmentation fault"). The error was ignored.

Ignoring errors was in fact By Design™; see commit 6358a64.

So what we need is for the `RunUnitTestApks` target to *not* report
errors, while still reporting errors.

Solve this conundrum by updating the `<RunInstrumentationTests/>` task
so that when the `nunit2-results-path` value can't be found, it
reports a *successful* run, and sets the
`RunInstrumentationTests.FailedToRun` property to the name of the
component that failed. This allows the `ReleaseAndroidTarget` target
to check for any failures, and report an `<Error/>` after shutting
down the emulator instance.

This should allow the build to "fail" when `nunit2-results-path` can't
be found -- a sign that the `.apk` unit tests didn't finish executing
-- while also allowing the emulator to be shut down.

Aside: `xbuild` doesn't support `ContinueOnError="ErrorAndContinue"`,
nor does it set `<Output/>` properties whenever a task fails.
Consequently, the `<RunInstrumentationTests/>` task *must* return true
when `nunit2-results-path` isn't found, because there's no other way
to convince `xbuild` to collect the `<Output/>` values.

[0]: dotnet#445
[1]: dotnet#474
@dellis1972 dellis1972 merged commit 3b893cd into dotnet:master Mar 9, 2017
@github-actions github-actions bot locked and limited conversation to collaborators Feb 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants