Flake and improve alert tests

[dgoodwin]

TRT has decided with input from @deads2k that we're ok with moving the generic alert backstop tests to always flake, as the signal is not as high value. The post-upgrade alert test fails 30% of the time globally, and the post-conformance variant is also not stellar, and both disproportionately affect certain NURPs.

In sippy as of openshift/sippy#685 we will begin tracking the reasons this test is flaking in the db by storing metadata about what alerts fired, so we'll have good insight into what may be causing issues, provided somebody remembers to look.

PR also re-introduces the refactor to merge the two backstop alert tests into a common code path, which they once were forked from. We thought this caused a regression, turned out it was something else, so this change should be good provided we start getting good payloads again after #27553.

Also improves output of the alert tests for better parsing in sippy. Clearly identifies if we accept or reject the failure, and if there is an associated bug.

[stbenjam]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/OWNERS [dgoodwin,stbenjam]
• test/OWNERS [dgoodwin,stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 403f0a4 and 2 for PR HEAD e1b50be in total

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 640432d and 1 for PR HEAD e1b50be in total

[openshift-ci]

@dgoodwin: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-single-node-upgrade	e1b50be	link	false	/test e2e-aws-ovn-single-node-upgrade
ci/prow/e2e-openstack-ovn	e1b50be	link	false	/test e2e-openstack-ovn
ci/prow/e2e-aws-ovn-cgroupsv2	e1b50be	link	false	/test e2e-aws-ovn-cgroupsv2
ci/prow/e2e-aws-ovn-single-node-serial	e1b50be	link	false	/test e2e-aws-ovn-single-node-serial
ci/prow/e2e-aws-ovn-single-node	e1b50be	link	false	/test e2e-aws-ovn-single-node
ci/prow/e2e-aws-ovn-upgrade	e1b50be	link	false	/test e2e-aws-ovn-upgrade

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[dgoodwin]

/override ci/prow/e2e-gcp-ovn-upgrade

The job just missed on pod sandbox.
Checked optional jobs, mostly aws install failures, which seem far more common than sippy indicates they are lately.

[openshift-ci]

@dgoodwin: Overrode contexts on behalf of dgoodwin: ci/prow/e2e-gcp-ovn-upgrade

Details

In response to this:

/override ci/prow/e2e-gcp-ovn-upgrade

The job just missed on pod sandbox.
Checked optional jobs, mostly aws install failures, which seem far more common than sippy indicates they are lately.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.