Write test failure summary, and add new command to request failure risk analysis from sippy. #27486

dgoodwin · 2022-10-21T14:47:09Z

This change begins writing out a brief test failure summary artifact in a format that is logical and also deserializable by sippy.

It also adds a new sub-command that can be used to merge several of these files, submit them to sippy, and write the resulting risk analysis to the artifacts directory.

Requires openshift/sippy#657

TRT-602

dgoodwin · 2022-10-25T11:16:12Z

Looks to be working, see the two test-flakes-and-failures files in: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/27486/pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade/1584599588365406208/artifacts/e2e-aws-ovn-single-node-upgrade/single-node-e2e-test/artifacts/junit/

I will remove the extraneous - before the suffix.

dgoodwin · 2022-10-26T17:46:13Z

Depends on openshift/sippy#657

/hold

This will be used to submit to a new sippy risk analysis API to determine how risky the test failures in this job run are. As with junit xmls, there can be multiple as we often invoke openshift-tests multiple times in one job run. Includes a change to make timestamp suffixes on junit artifacts more consistent.

… and upload. This merges test-flake files, submits them to sippy, and writes the returned risk analysis as an additional artifact. It is intended to be called from the workflow steps that run openshift-tests in the release repo.

neisw · 2022-10-31T14:20:09Z

pkg/riskanalysis/cmd.go

+	}
+
+	// TODO: query sippy
+	url := "http://localhost:8080/api/jobs/runs/risk_analysis"


Would we want this to be a config option so we could change down the road without updating the code?

Oops, that should be the real URL.

If sippy moves we'll have to make a commit somewhere, would that be better in release repo to update config, or in origin? Release repo might be a little better as it's probably exempt from code freeze. I could do a CLI flag for this with a default, so it's there if we ever need it.

Having it in the test step (maybe with a default here) makes sense. If it ever changes we'll avoid having to backport the changes

I like the idea of cli flag that can be defaulted and then you can change it in release repo when you call this as a step. Also makes it nice so you can call it locally for testing.

DennisPeriquet · 2022-10-31T14:59:14Z

pkg/riskanalysis/write_test_failure_summary.go

+
+	jr := ProwJobRun{
+		ProwJob: ProwJob{Name: os.Getenv("JOB_NAME")},
+		URL:     os.Getenv("JOB_URL"), // just a guess, may not exist


In this sample podinfo.json from this prow job, there is no JOB_URL. It does have BUILD_ID if you wanted to construct an URL later.

Thanks, I will get this updated in my followup.

xueqzhan · 2022-10-31T15:02:42Z

Looking good to me except the Sippy URL string.

DennisPeriquet · 2022-10-31T15:18:00Z

/lgtm

stbenjam · 2022-10-31T16:16:40Z

/hold cancel

stbenjam · 2022-10-31T16:17:29Z

/hold

Sippy URL needs fixing

dgoodwin · 2022-10-31T16:37:25Z

URL fixed, I will follow up with the CLI arg.

stbenjam · 2022-10-31T17:28:50Z

/lgtm

dgoodwin · 2022-10-31T18:18:36Z

Apologies, I'm going to have to bother folks for another lgtm anyhow, I made the requested modifications in this PR.

stbenjam · 2022-10-31T18:37:31Z

/lgtm

openshift-ci · 2022-10-31T18:37:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DennisPeriquet, dgoodwin, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~cmd/OWNERS~~ [dgoodwin,stbenjam]
~~pkg/OWNERS~~ [dgoodwin,stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dgoodwin · 2022-11-01T11:10:27Z

/hold cancel
/retest

dgoodwin · 2022-11-01T14:06:18Z

/override ci/prow/e2e-gcp-ovn-upgrade

Failing on disruption.

openshift-ci · 2022-11-01T14:08:26Z

@dgoodwin: Overrode contexts on behalf of dgoodwin: ci/prow/e2e-gcp-ovn-upgrade

Details

In response to this:

/override ci/prow/e2e-gcp-ovn-upgrade

Failing on disruption.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-11-01T14:13:00Z

@dgoodwin: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-csi	`b3a776a`	link	false	`/test e2e-aws-csi`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

dgoodwin · 2022-11-03T12:17:21Z

/cherry-pick release-4.12

openshift-cherrypick-robot · 2022-11-03T12:20:49Z

@dgoodwin: new pull request created: #27520

Details

In response to this:

/cherry-pick release-4.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dgoodwin changed the title ~~write test failures~~ WIP: write test failures Oct 21, 2022

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 21, 2022

openshift-ci bot requested review from csrwng and jwforres October 21, 2022 14:48

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 21, 2022

openshift-merge-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 22, 2022

dgoodwin force-pushed the write-test-failures branch from 8843bf4 to cb8d796 Compare October 26, 2022 17:43

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 26, 2022

dgoodwin changed the title ~~WIP: write test failures~~ Write test failure summary, and add new command to request failure risk analysis from sippy. Oct 26, 2022

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 26, 2022

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 26, 2022

dgoodwin force-pushed the write-test-failures branch from cb8d796 to 296e32d Compare October 26, 2022 17:49

openshift-merge-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 26, 2022

dgoodwin added 3 commits October 31, 2022 07:54

Clarify summary does not yet contain flakes.

19740f0

dgoodwin force-pushed the write-test-failures branch from 296e32d to 19740f0 Compare October 31, 2022 10:57

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 31, 2022

neisw reviewed Oct 31, 2022

View reviewed changes

DennisPeriquet reviewed Oct 31, 2022

View reviewed changes

openshift-ci bot assigned DennisPeriquet Oct 31, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2022

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 31, 2022

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 31, 2022

Fix to use actual sippy api instead of localhost.

8eb7f5f

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2022

openshift-ci bot assigned stbenjam Oct 31, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2022

dgoodwin added 2 commits October 31, 2022 15:06

Add optional CLI arg for sippy risk analysis API endpoint.

5d659aa

Add prow job run IDs to submitted request body.

b3a776a

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2022

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 1, 2022

openshift-merge-robot merged commit 8451d66 into openshift:master Nov 1, 2022

dgoodwin deleted the write-test-failures branch November 3, 2022 12:17

openshift-cherrypick-robot mentioned this pull request Nov 3, 2022

[release-4.12] Write test failure summary, and add new command to request failure risk analysis from sippy. #27520

Closed

Write test failure summary, and add new command to request failure risk analysis from sippy. #27486

Write test failure summary, and add new command to request failure risk analysis from sippy. #27486

Uh oh!

Conversation

dgoodwin commented Oct 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dgoodwin commented Oct 25, 2022

Uh oh!

dgoodwin commented Oct 26, 2022

Uh oh!

neisw Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

dgoodwin Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

stbenjam Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

DennisPeriquet Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

DennisPeriquet Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

dgoodwin Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

xueqzhan commented Oct 31, 2022

Uh oh!

DennisPeriquet commented Oct 31, 2022

Uh oh!

stbenjam commented Oct 31, 2022

Uh oh!

stbenjam commented Oct 31, 2022

Uh oh!

dgoodwin commented Oct 31, 2022

Uh oh!

stbenjam commented Oct 31, 2022

Uh oh!

dgoodwin commented Oct 31, 2022

Uh oh!

stbenjam commented Oct 31, 2022

Uh oh!

openshift-ci bot commented Oct 31, 2022

Uh oh!

dgoodwin commented Nov 1, 2022

Uh oh!

dgoodwin commented Nov 1, 2022

Uh oh!

openshift-ci bot commented Nov 1, 2022

Uh oh!

openshift-ci bot commented Nov 1, 2022

Uh oh!

dgoodwin commented Nov 3, 2022

Uh oh!

openshift-cherrypick-robot commented Nov 3, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dgoodwin commented Oct 21, 2022 •

edited

Loading