OCPBUGS-3633: Fix flake reporting for certain tests. #27553
Conversation
dgoodwin
changed the title
openshift-ci-robot
added
jira/valid-bug
|
@dgoodwin: This pull request references Jira Issue OCPBUGS-3633, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/hold |
openshift-ci
bot
added
the
do-not-merge/hold
|
@dgoodwin: trigger 4 job(s) of type blocking for the ci release of OCP 4.13
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/167ce980-64fa-11ed-9d34-9258f548db56-0 trigger 6 job(s) of type blocking for the nightly release of OCP 4.13
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/167ce980-64fa-11ed-9d34-9258f548db56-1 |
openshift-ci
bot
added
the
approved
|
/lgtm |
|
/hold Payloads look totally busted, not sure why yet. |
|
/payload 4.13 ci blocking |
|
@dgoodwin: trigger 4 job(s) of type blocking for the ci release of OCP 4.13
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/8bfdbc10-650f-11ed-9bdb-7ae30ba4a890-0 trigger 6 job(s) of type blocking for the nightly release of OCP 4.13
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/8bfdbc10-650f-11ed-9bdb-7ae30ba4a890-1 |
|
I don't think this worked. The payload command is pretty busted at least for minor upgrades, but this micro ran: In here we see no hit for the "firing or pending", and checking the junit files I can see it ran in the output, but it has no individual junit just like before. |
|
I went to that job and saw |
In PR openshift#27516 we suspect reporting of flakes broke due to a missed assumption that test.flake accompanied test.success. Our new goal is to more clearly have just one status set, so we're going to lean into the new approach and properly break out the flake state into it's own case.
dgoodwin
force-pushed
the
fix-flake-reporting
branch
from
47f277e to
aa7fb74
Compare
|
@xueqzhan may have found the issue, PR updated, trying again. /payload 4.13 ci blocking |
|
@dgoodwin: trigger 4 job(s) of type blocking for the ci release of OCP 4.13
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/42760470-65b5-11ed-9360-8bb1c6594d2c-0 |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: DennisPeriquet, dgoodwin, xueqzhan The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
||
| // calculate the effective test set we ran, excluding any incompletes | ||
| tests, _ = splitTests(tests, func(t *testCase) bool { return t.success || t.failed || t.skipped }) | ||
| tests, _ = splitTests(tests, func(t *testCase) bool { return t.success || t.flake || t.failed || t.skipped }) |
There was a problem hiding this comment.
Whatever this does, we should clean this up sometime this quarter.
|
lgtm as well. |
|
Flakes appeared in payload run for "failing or pending" alerts: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/openshift-origin-27553-ci-4.13-e2e-aws-ovn-upgrade/1592877200246312960 |
|
Payload itself didn't do well due to the minor upgrades failing until we get openshift/release#34072 Serial didn't install. And https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/aggregator-periodic-ci-openshift-release-master-ci-4.13-e2e-aws-ovn-upgrade/1592877204990070784 looks to have failed for real reasons. Test pass/flake/skip counts look correct, I think we've got it fixed. |
|
/cherry-pick release-4.12 /retest-required |
|
@dgoodwin: once the present PR merges, I will cherry-pick it on top of release-4.12 in a new PR and assign it to you. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
|
/test e2e-aws-ovn-fips |
|
@dgoodwin: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
@dgoodwin: Some pull requests linked via external trackers have merged: The following pull requests linked via external trackers have not merged:
These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-3633 has not been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@dgoodwin: new pull request created: #27557 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
7 participants
In PR #27516 we suspect reporting of flakes broke due to a missed
assumption that test.flake accompanied test.success. Our new goal is to
more clearly have just one status set, so we're going to lean into the
new approach and properly break out the flake state into it's own case.