Mark WorkItemExecution as passed when real test results exist#16577
Closed
Mark WorkItemExecution as passed when real test results exist#16577
Conversation
When a Helix work item fails and the test reporter has already posted real test results to the AzDO test run, the synthetic WorkItemExecution result is redundant. Marking it as failed prevents Build Analysis from going green even when all real test failures are matched to known issues. Before creating the synthetic result, query the test run for existing results from this work item (using automatedTestStorage). If results exist, mark WorkItemExecution as passed instead of failed. Work items that crash without producing any test results still get a failed WorkItemExecution entry — that's the case it was designed for. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
I'm not 100% sure this is the correct resolution to the problem but think it is worth brining up. |
Member
Isn't it either fix here or fix in BA? |
Member
Author
yeah, the alternative is working around it in BA |
Member
Author
|
I'm going to think on this a bit more |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a Helix work item fails and the test reporter has already posted real test results to the AzDO test run, the synthetic
WorkItemExecutionresult is redundant. Marking it as failed prevents Build Analysis from going green even when all real test failures are matched to known issues.Problem
CreateTestsForWorkItemscreates a synthetic.WorkItemExecutiontest result for every work item. If the work item failed, this result is marked asFailedwith a generic error message: "The Helix Work Item failed. Often this is due to a test crash."Build Analysis matches known-issue patterns against test
ErrorMessageandStackTracefields. The real test failures (e.g.NoOpRebuildtiming out) get matched to known issues, but the syntheticWorkItemExecutionentry has a generic message that doesn't match — so BA stays red.Example: dotnet/runtime#125373 — BA correctly matched 3 test failures to known issues (#116697, #124438), but the
RebuildTests.WorkItemExecutionentry kept BA red.Fix
Before creating the synthetic result, query the test run for existing results from this work item (using
automatedTestStorage). If results exist, markWorkItemExecutionas passed — the real results already capture the failure details.Work items that crash without producing any test results still get a failed
WorkItemExecutionentry — that's the case it was designed for.Trade-offs
$top=1keeps it lightweight)WorkItemExecutionentry for genuine crashes with no test output