Bug 1797244: test/extended/dr: update test to reflect new DR process by hexfusion · Pull Request #25071 · openshift/origin

hexfusion · 2020-06-05T22:30:00Z

The addition of cluster etcd operator introduced a new disaster recovery workflow that broke the disruptive tests. The previous workflow involved doing a snapshot restore of the etcd data file. The advantage to this approach is that once etcd starts the process is complete. The downside is a restore action must take place on each master node.

In 4.4+ we automate this process and require only a single invocation of the cluster-backup.sh script and cluster-restore.sh script. Given new machines the operator then scales etcd on the new machines resulting in a 3 node etcd cluster.

The downside of this process is that it is more disruptive as etcd and the rest of the control plane scales. For this reason, we needed to adjust not only some of the timings of the old test allowing for certain steps to take longer. But also to allow for transient etcd timeouts. During scaling from 1 to 2 etcd members we must lose quorum. The resulting leader election will result in client timeouts.

This is a starting place where we can begin to harden the requirements and improve. But for now lets make sure the workflow itself completes as expected.

openshift-ci-robot · 2020-06-05T22:30:01Z

@hexfusion: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

[wip] test/extended/dr: update test to reflect new DR process

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hexfusion · 2020-06-05T23:53:12Z

/test e2e-aws-disruptive

hexfusion · 2020-06-10T01:30:52Z

/test e2e-aws-disruptive

hexfusion · 2020-06-10T03:36:16Z

/test e2e-aws-disruptive

hexfusion · 2020-06-10T22:10:26Z

/test e2e-aws-disruptive

hexfusion · 2020-06-10T23:09:22Z

flake
/test e2e-aws-disruptive

hexfusion · 2020-06-11T01:52:00Z

[sig-etcd][Feature:DisasterRecovery][Disruptive] [Feature:EtcdRecovery] Cluster should restore itself after quorum loss [Disabled:Broken] [Serial] [Suite:openshift]

Passed \o/

test failed because of what I thought was #25087 but CI merged already

hexfusion · 2020-06-11T10:05:58Z

/test e2e-aws-disruptive

openshift-ci-robot · 2020-06-11T12:42:09Z

@hexfusion: This pull request references Bugzilla bug 1797244, which is invalid:

expected Bugzilla bug 1797244 to depend on a bug in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1797244: [wip] test/extended/dr: update test to reflect new DR process

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hexfusion · 2020-06-15T01:29:54Z

/test e2e-aws-disruptive

hexfusion · 2020-06-15T18:39:04Z

/test e2e-aws-disruptive

smarterclayton · 2020-06-17T14:16:30Z

/lgtm

openshift-ci-robot · 2020-06-17T14:19:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hexfusion, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~test/extended/OWNERS~~ [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-06-17T14:43:52Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-06-17T16:14:48Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-06-17T16:27:47Z

/retest

Please review the full test history for this PR and help us cut down flakes.

hexfusion · 2020-06-17T16:51:58Z

/bugzilla refresh

openshift-ci-robot · 2020-06-17T16:52:00Z

@hexfusion: This pull request references Bugzilla bug 1797244, which is invalid:

expected Bugzilla bug 1797244 to depend on a bug in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but no dependents were found

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-06-17T17:10:49Z

@hexfusion: This pull request references Bugzilla bug 1797244, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1797244: test/extended/dr: update test to reflect new DR process

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hexfusion · 2020-06-17T17:11:56Z

/refresh

hexfusion · 2020-06-17T17:22:10Z

/cherry-pick release-4.5

openshift-cherrypick-robot · 2020-06-17T17:22:11Z

@hexfusion: once the present PR merges, I will cherry-pick it on top of release-4.5 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2020-06-17T17:59:16Z

/retest

Please review the full test history for this PR and help us cut down flakes.

hexfusion · 2020-06-17T18:32:17Z

/test e2e-cmd

openshift-ci-robot · 2020-06-17T19:53:27Z

@hexfusion: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-conformance-k8s	`051ca88`	link	`/test e2e-conformance-k8s`
ci/prow/e2e-aws-jenkins	`051ca88`	link	`/test e2e-aws-jenkins`
ci/prow/e2e-aws	`051ca88`	link	`/test e2e-aws`
ci/prow/e2e-vsphere	`051ca88`	link	`/test e2e-vsphere`
ci/prow/launch-vsphere	`051ca88`	link	`/test launch-vsphere`
ci/prow/e2e-gcp-builds	`69ee373`	link	`/test e2e-gcp-builds`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2020-06-17T19:55:43Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-06-17T20:22:14Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-06-18T00:05:33Z

@hexfusion: All pull requests linked via external trackers have merged: openshift/origin#25071. Bugzilla bug 1797244 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1797244: test/extended/dr: update test to reflect new DR process

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2020-06-18T00:08:45Z

@hexfusion: new pull request created: #25147

Details

In response to this:

/cherry-pick release-4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

retroflexer · 2020-06-23T16:12:47Z

/cherry-pick release-4.4

openshift-cherrypick-robot · 2020-06-23T16:15:58Z

@retroflexer: #25071 failed to apply on top of branch "release-4.4":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	test/extended/dr/common.go
M	test/extended/dr/quorum_restore.go
Falling back to patching base and 3-way merge...
Auto-merging test/extended/dr/quorum_restore.go
CONFLICT (content): Merge conflict in test/extended/dr/quorum_restore.go
Auto-merging test/extended/dr/common.go
CONFLICT (content): Merge conflict in test/extended/dr/common.go
Patch failed at 0001 test/extended/dr: update test to reflect new DR process

Details

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 5, 2020

openshift-ci-robot requested review from adambkaplan and bparees June 5, 2020 22:33

hexfusion changed the base branch from release-4.5 to release-4.6 June 8, 2020 21:40

hexfusion force-pushed the fix-dr-tests branch 3 times, most recently from b197579 to 7b7a905 Compare June 10, 2020 00:59

hexfusion force-pushed the fix-dr-tests branch from 7b7a905 to 3d7160b Compare June 10, 2020 21:57

hexfusion changed the title ~~[wip] test/extended/dr: update test to reflect new DR process~~ Bug 1797244: [wip] test/extended/dr: update test to reflect new DR process Jun 11, 2020

openshift-ci-robot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jun 11, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 11, 2020

hexfusion changed the title ~~Bug 1797244: [wip] test/extended/dr: update test to reflect new DR process~~ [wip] Bug 1797244: test/extended/dr: update test to reflect new DR process Jun 11, 2020

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 11, 2020

hexfusion force-pushed the fix-dr-tests branch from 3d7160b to 63a408b Compare June 15, 2020 18:37

smarterclayton reviewed Jun 15, 2020

View reviewed changes

Comment thread test/extended/dr/quorum_restore.go

smarterclayton reviewed Jun 15, 2020

View reviewed changes

Comment thread test/extended/dr/quorum_restore.go Outdated

smarterclayton reviewed Jun 15, 2020

View reviewed changes

Comment thread test/extended/dr/quorum_restore.go Outdated

smarterclayton reviewed Jun 15, 2020

View reviewed changes

Comment thread test/extended/dr/quorum_restore.go Outdated

openshift-ci-robot assigned smarterclayton Jun 17, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 17, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 17, 2020

hexfusion changed the base branch from release-4.6 to master June 17, 2020 17:10

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jun 17, 2020

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 17, 2020

openshift-merge-robot merged commit c5076c9 into openshift:master Jun 18, 2020

openshift-cherrypick-robot mentioned this pull request Jun 18, 2020

[release-4.5] Bug 1845291: test/extended/dr: update test to reflect new DR process #25147

Merged

retroflexer mentioned this pull request Jun 23, 2020

[wip]: test/extended/dr: update test to reflect new DR process #25193

Closed

hexfusion deleted the fix-dr-tests branch June 23, 2020 20:40

Conversation

hexfusion commented Jun 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 5, 2020

Uh oh!

hexfusion commented Jun 5, 2020

Uh oh!

hexfusion commented Jun 10, 2020

Uh oh!

hexfusion commented Jun 10, 2020

Uh oh!

hexfusion commented Jun 10, 2020

Uh oh!

hexfusion commented Jun 10, 2020

Uh oh!

hexfusion commented Jun 11, 2020

Uh oh!

hexfusion commented Jun 11, 2020

Uh oh!

openshift-ci-robot commented Jun 11, 2020

Uh oh!

hexfusion commented Jun 15, 2020

Uh oh!

hexfusion commented Jun 15, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smarterclayton commented Jun 17, 2020

Uh oh!

openshift-ci-robot commented Jun 17, 2020

Uh oh!

openshift-bot commented Jun 17, 2020

Uh oh!

openshift-bot commented Jun 17, 2020

Uh oh!

openshift-bot commented Jun 17, 2020

Uh oh!

hexfusion commented Jun 17, 2020

Uh oh!

openshift-ci-robot commented Jun 17, 2020

Uh oh!

openshift-ci-robot commented Jun 17, 2020

Uh oh!

hexfusion commented Jun 17, 2020

Uh oh!

hexfusion commented Jun 17, 2020

Uh oh!

openshift-cherrypick-robot commented Jun 17, 2020

Uh oh!

openshift-bot commented Jun 17, 2020

Uh oh!

hexfusion commented Jun 17, 2020

Uh oh!

openshift-ci-robot commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Jun 17, 2020

Uh oh!

openshift-bot commented Jun 17, 2020

Uh oh!

openshift-ci-robot commented Jun 18, 2020

Uh oh!

openshift-cherrypick-robot commented Jun 18, 2020

Uh oh!

retroflexer commented Jun 23, 2020

Uh oh!

openshift-cherrypick-robot commented Jun 23, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hexfusion commented Jun 5, 2020 •

edited

Loading

openshift-ci-robot commented Jun 17, 2020 •

edited

Loading