Bug 1986453: Check for API server and node versions skew by QiWang19 · Pull Request #2658 · openshift/machine-config-operator

QiWang19 · 2021-07-02T17:38:39Z

ref: https://issues.redhat.com/browse/OCPNODE-595
replace #2552

- What I did

Check for API server and node versions skew.
Update with the message the Kube API version is skew too far, but do not force
Upgradeable=False according to enhancement https://github.com/openshift/enhancements/pull/762/files

- How to verify it

- Description for the changelog

QiWang19 · 2021-07-02T17:39:01Z

/assign @sinnykumari

kikisdeliveryservice

some initial questions/comments

kikisdeliveryservice

a few more comments

/hold

kikisdeliveryservice · 2021-07-02T23:14:54Z

@QiWang19 I know that this PR is based on someone else's previous PR, but I think we can refine it. What most of my comments are asking for is: using that work as a baseline, how can we make this into an understandable & actionable status for a user. Also keep in mind a pool can be large, so the info needs to be easy to consume if we have, say, 50 nodes that have unsupported skew.

Users can get into this situation bc a pool is paused and so it did not upgrade to match the apiserver and is still at an older version. They will likely have to let those pools upgrade (to at a minimum a supported skew) before they can initiate another clusterwide upgrade (which would cause the kubeapiserver to get even further away). So we need to think about telling them the state in a meaningful way, but also give them some hint about what they need to do to remedy it.

Happy to discuss further if you have any questions. =)

QiWang19 · 2021-07-06T20:29:19Z

@kikisdeliveryservice Thanks for the explanation. I have cleaned up some reviews. PTAL.

QiWang19 · 2021-07-07T14:40:35Z

/retest

QiWang19 · 2021-07-08T15:48:04Z

/retest

QiWang19 · 2021-07-09T15:54:39Z

/retest

QiWang19 · 2021-07-13T13:51:21Z

@kikisdeliveryservice Thanks for the explanation. I have cleaned up some reviews. Could you PTAL?

kikisdeliveryservice

a few more comments

kikisdeliveryservice · 2021-07-15T21:12:33Z

Just noting that when this is done, we need to get the commits updated so Qi is a co-author.

rphillips · 2021-07-27T19:33:06Z

I tested this PR locally and it does report the skew, and go back to no skew as expected.

kikisdeliveryservice

one last thing otherwise looks good

Co-authored-by: Qi Wang <qiwan@redhat.com> Signed-off-by: Qi Wang <qiwan@redhat.com>

kikisdeliveryservice

Thanks for all of the work on this. I think we've gotten it to a good state.

rphillips · 2021-07-27T20:03:34Z

/lgtm

openshift-ci · 2021-07-27T20:03:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kikisdeliveryservice, QiWang19, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kikisdeliveryservice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-07-27T21:05:41Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-07-27T22:00:43Z

@QiWang19: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-disruptive	61c86bf43e472ef41b55fc826940026bebabb547	link	`/test e2e-aws-disruptive`
ci/prow/okd-e2e-aws	`1471d2c`	link	`/test okd-e2e-aws`
ci/prow/e2e-aws-upgrade-single-node	`1471d2c`	link	`/test e2e-aws-upgrade-single-node`
ci/prow/e2e-aws-workers-rhel7	`1471d2c`	link	`/test e2e-aws-workers-rhel7`
ci/prow/e2e-ovn-step-registry	`1471d2c`	link	`/test e2e-ovn-step-registry`
ci/prow/e2e-aws-techpreview-featuregate	`1471d2c`	link	`/test e2e-aws-techpreview-featuregate`
ci/prow/e2e-aws-serial	`1471d2c`	link	`/test e2e-aws-serial`
ci/prow/e2e-vsphere-upgrade	`1471d2c`	link	`/test e2e-vsphere-upgrade`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci · 2021-07-27T22:21:51Z

@QiWang19: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#2658

Bugzilla bug 1986453 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1986453: Check for API server and node versions skew

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

The kubelet skew guards are from 1471d2c (Bug 1986453: Check for API server and node versions skew, 2021-07-27, openshift#2658). But the Kube API server also landed similar guards in openshift/cluster-kube-apiserver-operator@9ce4f74775 (add KubeletVersionSkewController, 2021-08-26, openshift/enhancements@0ba744e750 (eus-upgrades-mvp: don't enforce skew check in MCO, 2021-04-29, openshift/enhancements#762) had shifted the proposal form MCO-guards to KAS-guards, so I'm not entirely clear on why the MCO guards landed at all. But it's convenient for me that they did, because while I'm dropping them here, I'm recycling the Node lister for a new check. 4.19 is dropping bare-RHEL support, and I want the Node lister to look for RHEL entries like: osImage: Red Hat Enterprise Linux 8.6 (Ootpa) but we are ok with RHCOS entries like: osImage: Red Hat Enterprise Linux CoreOS 419.96.202503032242-0

The kubelet skew guards are from 1471d2c (Bug 1986453: Check for API server and node versions skew, 2021-07-27, openshift#2658). But the Kube API server also landed similar guards in openshift/cluster-kube-apiserver-operator@9ce4f74775 (add KubeletVersionSkewController, 2021-08-26, openshift/cluster-kube-apiserver-operator#1199). openshift/enhancements@0ba744e750 (eus-upgrades-mvp: don't enforce skew check in MCO, 2021-04-29, openshift/enhancements#762) had shifted the proposal form MCO-guards to KAS-guards, so I'm not entirely clear on why the MCO guards landed at all. But it's convenient for me that they did, because while I'm dropping them here, I'm recycling the Node lister for a new check. 4.19 is dropping bare-RHEL support, and I want the Node lister to look for RHEL entries like: osImage: Red Hat Enterprise Linux 8.6 (Ootpa) but we are ok with RHCOS entries like: osImage: Red Hat Enterprise Linux CoreOS 419.96.202503032242-0

The kubelet skew guard is from 1471d2c (Bug 1986453: Check for API server and node versions skew, 2021-07-27, openshift#2658). But the Kube API server also landed a similar guard in openshift/cluster-kube-apiserver-operator@9ce4f74775 (add KubeletVersionSkewController, 2021-08-26, openshift/cluster-kube-apiserver-operator#1199). openshift/enhancements@0ba744e750 (eus-upgrades-mvp: don't enforce skew check in MCO, 2021-04-29, openshift/enhancements#762) had shifted the proposal from MCO-guards to KAS-guards, so I'm not clear on why the MCO guard landed. This commit drops it, to consolidate around the KAS-side guard.

The kubelet skew guards are from 1471d2c (Bug 1986453: Check for API server and node versions skew, 2021-07-27, openshift#2658). But the Kube API server also landed similar guards in openshift/cluster-kube-apiserver-operator@9ce4f74775 (add KubeletVersionSkewController, 2021-08-26, openshift/cluster-kube-apiserver-operator#1199). openshift/enhancements@0ba744e750 (eus-upgrades-mvp: don't enforce skew check in MCO, 2021-04-29, openshift/enhancements#762) had shifted the proposal form MCO-guards to KAS-guards, so I'm not entirely clear on why the MCO guards landed at all. But it's convenient for me that they did, because while I'm dropping them here, I'm recycling the Node lister for a new check. 4.19 is dropping bare-RHEL support, and I want the Node lister to look for RHEL entries like: osImage: Red Hat Enterprise Linux 8.6 (Ootpa) but we are ok with RHCOS entries like: osImage: Red Hat Enterprise Linux CoreOS 419.96.202503032242-0

The kubelet skew guard is from 1471d2c (Bug 1986453: Check for API server and node versions skew, 2021-07-27, openshift#2658). But the Kube API server also landed a similar guard in openshift/cluster-kube-apiserver-operator@9ce4f74775 (add KubeletVersionSkewController, 2021-08-26, openshift/cluster-kube-apiserver-operator#1199). openshift/enhancements@0ba744e750 (eus-upgrades-mvp: don't enforce skew check in MCO, 2021-04-29, openshift/enhancements#762) had shifted the proposal from MCO-guards to KAS-guards, so I'm not clear on why the MCO guard landed. This commit drops it, to consolidate around the KAS-side guard.

The kubelet skew guards are from 1471d2c (Bug 1986453: Check for API server and node versions skew, 2021-07-27, openshift#2658). But the Kube API server also landed similar guards in openshift/cluster-kube-apiserver-operator@9ce4f74775 (add KubeletVersionSkewController, 2021-08-26, openshift/cluster-kube-apiserver-operator#1199). openshift/enhancements@0ba744e750 (eus-upgrades-mvp: don't enforce skew check in MCO, 2021-04-29, openshift/enhancements#762) had shifted the proposal form MCO-guards to KAS-guards, so I'm not entirely clear on why the MCO guards landed at all. But it's convenient for me that they did, because while I'm dropping them here, I'm recycling the Node lister for a new check. 4.19 is dropping bare, package-managed RHEL support. I'd initially thought about looking for RHEL entries like: osImage: Red Hat Enterprise Linux 8.6 (Ootpa) while excluding RHCOS entries like: osImage: Red Hat Enterprise Linux CoreOS 419.96.202503032242-0 But instead of switching on osImage, I'm using the node.openshift.io/os_id label to find package-managed RHEL Nodes. The machine-config operator is setting up the label [1] based on the ID value in /etc/os-release. On RHCOS instances, the ID value is 'rhcos' [2]. On package-managed RHEL, it's 'rhel' [3,4]. [1]: https://github.com/openshift/machine-config-operator/blob/ddc18e84f4a0650e0e87aa0a4f90f9cf01b5259c/templates/worker/01-worker-kubelet/_base/units/kubelet.service.yaml#L19-L31 [2]: https://github.com/openshift/os/blob/41f6a028d37b750db0bf4257447d809bd9cbe4bf/manifest-ocp-rhel-9.6.yaml#L41 [3]: https://github.com/openshift/enhancements/blob/ea465e192bfb58ec8654f1c904a4af68777f68ec/enhancements/rhcos/split-rhcos-into-layers.md?plain=1#L416 [4]: https://github.com/openshift/machine-config-operator/blob/ddc18e84f4a0650e0e87aa0a4f90f9cf01b5259c/pkg/daemon/osrelease/osrelease.go#L69

The kubelet skew guard is from 1471d2c (Bug 1986453: Check for API server and node versions skew, 2021-07-27, openshift#2658). But the Kube API server also landed a similar guard in openshift/cluster-kube-apiserver-operator@9ce4f74775 (add KubeletVersionSkewController, 2021-08-26, openshift/cluster-kube-apiserver-operator#1199). openshift/enhancements@0ba744e750 (eus-upgrades-mvp: don't enforce skew check in MCO, 2021-04-29, openshift/enhancements#762) had shifted the proposal from MCO-guards to KAS-guards, so I'm not clear on why the MCO guard landed. This commit drops it, to consolidate around the KAS-side guard.

openshift-ci Bot requested review from kikisdeliveryservice and sinnykumari July 2, 2021 17:38

openshift-ci Bot assigned sinnykumari Jul 2, 2021

kikisdeliveryservice reviewed Jul 2, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

kikisdeliveryservice unassigned sinnykumari Jul 2, 2021

kikisdeliveryservice requested a review from rphillips July 2, 2021 21:02

kikisdeliveryservice reviewed Jul 2, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 2, 2021

kikisdeliveryservice reviewed Jul 2, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

kikisdeliveryservice reviewed Jul 2, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

kikisdeliveryservice reviewed Jul 2, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

kikisdeliveryservice changed the title ~~Check for API server and node versions skew~~ [OCPNODE-595] Check for API server and node versions skew Jul 3, 2021

QiWang19 force-pushed the no-skew-check branch 4 times, most recently from 4cf72ac to 3664ebe Compare July 6, 2021 20:25

kikisdeliveryservice reviewed Jul 13, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

Comment thread pkg/operator/status.go Outdated

QiWang19 force-pushed the no-skew-check branch from 3664ebe to a1569b1 Compare July 15, 2021 20:28

kikisdeliveryservice reviewed Jul 15, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

kikisdeliveryservice reviewed Jul 15, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

kikisdeliveryservice reviewed Jul 15, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

QiWang19 force-pushed the no-skew-check branch from a1569b1 to b8bbd14 Compare July 16, 2021 17:53

kikisdeliveryservice reviewed Jul 27, 2021

View reviewed changes

Comment thread pkg/operator/status.go Outdated

add kubelet skew check for MCO upgradeable

1471d2c

Co-authored-by: Qi Wang <qiwan@redhat.com> Signed-off-by: Qi Wang <qiwan@redhat.com>

QiWang19 force-pushed the no-skew-check branch from 61c86bf to 1471d2c Compare July 27, 2021 19:55

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 27, 2021

kikisdeliveryservice approved these changes Jul 27, 2021

View reviewed changes

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 27, 2021

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 27, 2021

openshift-merge-robot merged commit 9c07edd into openshift:master Jul 27, 2021

wking mentioned this pull request Mar 26, 2025

OCPBUGS-53427: pkg/operator/status: Drop kubelet skew guard, add RHEL guard #4956

Merged

wking mentioned this pull request Apr 3, 2025

OCPBUGS-54611: pkg/operator/status: Drop kubelet skew guard #4970

Merged

Conversation

QiWang19 commented Jul 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

QiWang19 commented Jul 2, 2021

Uh oh!

kikisdeliveryservice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kikisdeliveryservice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kikisdeliveryservice commented Jul 2, 2021

Uh oh!

QiWang19 commented Jul 6, 2021

Uh oh!

QiWang19 commented Jul 7, 2021

Uh oh!

QiWang19 commented Jul 8, 2021

Uh oh!

QiWang19 commented Jul 9, 2021

Uh oh!

QiWang19 commented Jul 13, 2021

Uh oh!

kikisdeliveryservice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kikisdeliveryservice commented Jul 15, 2021

Uh oh!

Uh oh!

rphillips commented Jul 27, 2021

Uh oh!

kikisdeliveryservice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kikisdeliveryservice left a comment

Choose a reason for hiding this comment

Uh oh!

rphillips commented Jul 27, 2021

Uh oh!

openshift-ci Bot commented Jul 27, 2021

Uh oh!

openshift-bot commented Jul 27, 2021

Uh oh!

openshift-ci Bot commented Jul 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Jul 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

QiWang19 commented Jul 2, 2021 •

edited

Loading

openshift-ci Bot commented Jul 27, 2021 •

edited

Loading