Skip to content

OTA-448: Add upgrade tests for a cluster with infra nodes #21175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
openshift-merge-robot merged 4 commits into from
Dec 6, 2021
Merged

OTA-448: Add upgrade tests for a cluster with infra nodes #21175

openshift-merge-robot merged 4 commits into from
Dec 6, 2021

Conversation

@vrutkovs
Copy link
Contributor

@vrutkovs vrutkovs commented Aug 18, 2021 •
edited
Loading

This workflow would create a new MachineSet based on "worker" machineset, add "infra" node role and move infra components (ingress, registry, monitoring) to these nodes.

The workflow is based on paused workers upgrade, so during upgrade:

  • infra pool in unpaused, worker pool is paused
  • upgrade starts, infra nodes are updated
  • worker pool is unpaused, workers are updated

TODO:

  • Omit 4.7 as it can't move monitoring pods to infra nodes?
  • Annotate infra MCP as required during update so that it would be treated separately from workers

Ref: OTA-448

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 18, 2021
@openshift-ci openshift-ci bot requested review from hasbro17 and joelsmith August 18, 2021 09:23
@openshift-ci openshift-ci bot added the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label Aug 18, 2021
@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch 6 times, most recently from f556464 to 89bb952 Compare August 18, 2021 13:14
@openshift-ci openshift-ci bot removed the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label Aug 18, 2021
@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch 8 times, most recently from 12fa2ad to 822114b Compare August 19, 2021 12:27
@vrutkovs
Copy link
Contributor Author

vrutkovs commented Aug 19, 2021

/test pj-rehearse

@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch from 822114b to 8dcb792 Compare August 19, 2021 13:50
@vrutkovs vrutkovs changed the title WIP Add 4.7 stable -> 4.8 upgrade test for a cluster with infra nodes Add upgrade tests for a cluster with infra nodes Aug 19, 2021
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 19, 2021
@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch from 8dcb792 to 3df261a Compare August 19, 2021 17:11
@vrutkovs
Copy link
Contributor Author

vrutkovs commented Aug 19, 2021

/cc @LalatenduMohanty @wking

@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch from 3df261a to 8f1138e Compare August 19, 2021 17:47
@vrutkovs
Copy link
Contributor Author

vrutkovs commented Aug 20, 2021

/retest

@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch from 38a3493 to c5d9807 Compare November 23, 2021 20:10
@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch 4 times, most recently from a9efb6f to 3e28e7e Compare November 30, 2021 12:41
@openshift-ci openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 30, 2021
@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch from 3e28e7e to 24e1cea Compare November 30, 2021 12:41
@openshift-ci openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 30, 2021
@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch from 24e1cea to 754553c Compare November 30, 2021 13:02
@openshift-ci openshift-ci bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 30, 2021
@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch from 754553c to 47f66a7 Compare November 30, 2021 13:05
@openshift-ci openshift-ci bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Nov 30, 2021
Vadim Rutkovsky added 3 commits November 30, 2021 18:24
workflows: add a new workflow creating infra nodes
53035df 
This workflow would create a new MachineSet based on "worker" machineset, add "infra" node role and move infra components (ingress, registry, monitoring) to these nodes
Add 4.8 stable -> 4.9 and 4.9->4.10 upgrade tests for a cluster with …
94d203d 
…infra nodes
inframachineset: ensure pods land on infra nodes
cd985bc
@vrutkovs vrutkovs force-pushed the upgrade-workflow-infra branch from 47f66a7 to cd985bc Compare November 30, 2021 17:24
@vrutkovs
Copy link
Contributor Author

vrutkovs commented Dec 1, 2021

/test app-ci-config-dry

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 1, 2021

@vrutkovs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade-rollback 54c6ff8222393c40364ced49407a41ef5cee12dc link /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-aws-upgrade-rollback 54c6ff8222393c40364ced49407a41ef5cee12dc link /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-upgrade-infra 47f66a774f2167c3c8e2f0eab8b1dc89c310cf84 link unknown /test pj-rehearse
ci/rehearse/periodic-ci-openshift-release-master-ci-4.9-upgrade-from-stable-4.8-e2e-aws-upgrade-infra 32b9cfc link unknown /test pj-rehearse
ci/prow/pj-rehearse 32b9cfc link false /test pj-rehearse

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

wking
wking approved these changes Dec 6, 2021
View reviewed changes
Copy link
Member

@wking wking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 6, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 6, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheesesashimi, vrutkovs, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit b661bfe into openshift:master Dec 6, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 6, 2021

@vrutkovs: Updated the following 3 configmaps:

  • ci-operator-master-configs configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-release-master__ci-4.10-upgrade-from-stable-4.9.yaml using file ci-operator/config/openshift/release/openshift-release-master__ci-4.10-upgrade-from-stable-4.9.yaml
    • key openshift-release-master__ci-4.9-upgrade-from-stable-4.8.yaml using file ci-operator/config/openshift/release/openshift-release-master__ci-4.9-upgrade-from-stable-4.8.yaml
  • job-config-master-periodics configmap in namespace ci at cluster app.ci using the following files:
    • key openshift-release-master-periodics.yaml using file ci-operator/jobs/openshift/release/openshift-release-master-periodics.yaml
  • step-registry configmap in namespace ci at cluster app.ci using the following files:
    • key OWNERS using file ci-operator/step-registry/ipi/conf/inframachineset/OWNERS
    • key ipi-conf-inframachineset-commands.sh using file ci-operator/step-registry/ipi/conf/inframachineset/ipi-conf-inframachineset-commands.sh
    • key ipi-conf-inframachineset-ref.metadata.json using file ci-operator/step-registry/ipi/conf/inframachineset/ipi-conf-inframachineset-ref.metadata.json
    • key ipi-conf-inframachineset-ref.yaml using file ci-operator/step-registry/ipi/conf/inframachineset/ipi-conf-inframachineset-ref.yaml
    • key OWNERS using file ci-operator/step-registry/openshift/upgrade/aws/infra/OWNERS
    • key openshift-upgrade-aws-infra-workflow.metadata.json using file ci-operator/step-registry/openshift/upgrade/aws/infra/openshift-upgrade-aws-infra-workflow.metadata.json
    • key openshift-upgrade-aws-infra-workflow.yaml using file ci-operator/step-registry/openshift/upgrade/aws/infra/openshift-upgrade-aws-infra-workflow.yaml
Details

In response to this:

This workflow would create a new MachineSet based on "worker" machineset, add "infra" node role and move infra components (ingress, registry, monitoring) to these nodes.

The workflow is based on paused workers upgrade, so during upgrade:

  • infra pool in unpaused, worker pool is paused
  • upgrade starts, infra nodes are updated
  • worker pool is unpaused, workers are updated

TODO:

  • Omit 4.7 as it can't move monitoring pods to infra nodes?
  • Annotate infra MCP as required during update so that it would be treated separately from workers

Ref: OTA-448

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@qiliRedHat qiliRedHat mentioned this pull request Dec 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hasbro17 hasbro17 Awaiting requested review from hasbro17

@joelsmith joelsmith Awaiting requested review from joelsmith

@LalatenduMohanty LalatenduMohanty Awaiting requested review from LalatenduMohanty

@cheesesashimi cheesesashimi Awaiting requested review from cheesesashimi

1 more reviewer

@wking wking wking approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@wking wking

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vrutkovs @cheesesashimi @wking @openshift-merge-robot