Skip to content

TRT-2578: Revert "OCPBUGS-74513: OCPBUGS-74519: OCPBUGS-78487: Remove MachineConfigNodes and PinnedImages feature gates and v1alpha1 references"#2767

Closed
smg247 wants to merge 1 commit intoopenshift:masterfrom
smg247:revert-2715-machine-config-nodes
Closed

Conversation

@smg247
Copy link
Copy Markdown
Member

@smg247 smg247 commented Mar 17, 2026

This reverts #2715

Why the revert?

This PR is causing blocking job failures (aggregated-aws-ovn-upgrade, aggregated-azure-ovn-upgrade, aggregated-gcp-ovn-upgrade) in the ci amd64 payload 4.22.0-0.ci-2026-03-17-030533.

Evidence

  • Removes MachineConfigNodes and PinnedImages feature gates. Azure upgrade job shows 50% failure rate with MCO upgrade failures, cluster readiness failures, and node readiness issues. Feature gate removal on 4.21->4.22 upgrade path is a high-risk change. All 3 aggregated jobs started failing in this payload. Strong temporal and component match.

Tracking

  • Revert JIRA: TRT-2578
  • Original PR JIRA: OCPBUGS-74513, OCPBUGS-74519, OCPBUGS-78487

To unrevert

/cc @isabella-janssen

To restore your change, please:

  1. Revert this revert PR
  2. Layer a fix commit on top addressing the CI failures
  3. Ensure the following jobs pass before merging:
    • aggregated-aws-ovn-upgrade-4.22-minor-release-openshift-release-analysis-aggregator
    • aggregated-azure-ovn-upgrade-4.22-minor-release-openshift-release-analysis-aggregator
    • aggregated-gcp-ovn-upgrade-4.22-micro-release-openshift-release-analysis-aggregator

🤖 Generated with Revertomatic

…gs-74513"

This reverts commit b3c6104, reversing
changes made to 26dbe2a.
@openshift-ci-robot
Copy link
Copy Markdown

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci openshift-ci Bot requested a review from isabella-janssen March 17, 2026 11:55
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 17, 2026
@openshift-ci-robot
Copy link
Copy Markdown

@smg247: This pull request references Jira Issue OCPBUGS-74513, which is invalid:

  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This reverts #2715

Why the revert?

This PR is causing blocking job failures (aggregated-aws-ovn-upgrade, aggregated-azure-ovn-upgrade, aggregated-gcp-ovn-upgrade) in the ci amd64 payload 4.22.0-0.ci-2026-03-17-030533.

Evidence

  • Removes MachineConfigNodes and PinnedImages feature gates. Azure upgrade job shows 50% failure rate with MCO upgrade failures, cluster readiness failures, and node readiness issues. Feature gate removal on 4.21->4.22 upgrade path is a high-risk change. All 3 aggregated jobs started failing in this payload. Strong temporal and component match.

Tracking

  • Revert JIRA: TRT-PENDING
  • Original PR JIRA: OCPBUGS-74513, OCPBUGS-74519, OCPBUGS-78487

To unrevert

/cc @isabella-janssen

To restore your change, please:

  1. Revert this revert PR
  2. Layer a fix commit on top addressing the CI failures
  3. Ensure the following jobs pass before merging:
  • aggregated-aws-ovn-upgrade-4.22-minor-release-openshift-release-analysis-aggregator
  • aggregated-azure-ovn-upgrade-4.22-minor-release-openshift-release-analysis-aggregator
  • aggregated-gcp-ovn-upgrade-4.22-micro-release-openshift-release-analysis-aggregator

🤖 Generated with Revertomatic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

Hello @smg247! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

@openshift-ci openshift-ci Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Mar 17, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Mar 17, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 1481c11a-72d5-4be5-a367-d2059596dba5

📥 Commits

Reviewing files that changed from the base of the PR and between 5c75e62 and b3678e8.

⛔ Files ignored due to path filters (10)
  • machineconfiguration/v1/zz_generated.featuregated-crd-manifests/machineconfignodes.machineconfiguration.openshift.io/MachineConfigNodes.yaml is excluded by !**/zz_generated.featuregated-crd-manifests/**
  • machineconfiguration/v1/zz_generated.featuregated-crd-manifests/machineconfigpools.machineconfiguration.openshift.io/AAA_ungated.yaml is excluded by !**/zz_generated.featuregated-crd-manifests/**
  • machineconfiguration/v1/zz_generated.featuregated-crd-manifests/machineconfigpools.machineconfiguration.openshift.io/OSStreams.yaml is excluded by !**/zz_generated.featuregated-crd-manifests/**
  • machineconfiguration/v1/zz_generated.featuregated-crd-manifests/machineconfigpools.machineconfiguration.openshift.io/PinnedImages.yaml is excluded by !**/zz_generated.featuregated-crd-manifests/**
  • machineconfiguration/v1/zz_generated.featuregated-crd-manifests/pinnedimagesets.machineconfiguration.openshift.io/PinnedImages.yaml is excluded by !**/zz_generated.featuregated-crd-manifests/**
  • machineconfiguration/v1alpha1/zz_generated.crd-manifests/0000_80_machine-config_01_machineconfignodes.crd.yaml is excluded by !**/zz_generated.crd-manifests/*
  • machineconfiguration/v1alpha1/zz_generated.crd-manifests/0000_80_machine-config_01_pinnedimagesets.crd.yaml is excluded by !**/zz_generated.crd-manifests/*
  • machineconfiguration/v1alpha1/zz_generated.featuregated-crd-manifests/machineconfignodes.machineconfiguration.openshift.io/MachineConfigNodes.yaml is excluded by !**/zz_generated.featuregated-crd-manifests/**
  • machineconfiguration/v1alpha1/zz_generated.featuregated-crd-manifests/pinnedimagesets.machineconfiguration.openshift.io/PinnedImages.yaml is excluded by !**/zz_generated.featuregated-crd-manifests/**
  • openapi/generated_openapi/zz_generated.openapi.go is excluded by !openapi/**
📒 Files selected for processing (27)
  • features.md
  • features/features.go
  • machineconfiguration/v1/tests/machineconfignodes.machineconfiguration.openshift.io/IrreconcilableMachineConfig.yaml
  • machineconfiguration/v1/tests/machineconfignodes.machineconfiguration.openshift.io/MachineConfigNodes.yaml
  • machineconfiguration/v1/tests/machineconfignodes.machineconfiguration.openshift.io/NoRegistryClusterInstall.yaml
  • machineconfiguration/v1/tests/machineconfigpools.machineconfiguration.openshift.io/PinnedImages.yaml
  • machineconfiguration/v1/tests/pinnedimagesets.machineconfiguration.openshift.io/PinnedImages.yaml
  • machineconfiguration/v1/types.go
  • machineconfiguration/v1/types_machineconfignode.go
  • machineconfiguration/v1/types_pinnedimageset.go
  • machineconfiguration/v1/zz_generated.featuregated-crd-manifests.yaml
  • machineconfiguration/v1alpha1/register.go
  • machineconfiguration/v1alpha1/tests/machineconfignodes.machineconfiguration.openshift.io/MachineConfigNodes.yaml
  • machineconfiguration/v1alpha1/tests/pinnedimagesets.machineconfiguration.openshift.io/PinnedImages.yaml
  • machineconfiguration/v1alpha1/types_machineconfignode.go
  • machineconfiguration/v1alpha1/types_pinnedimageset.go
  • machineconfiguration/v1alpha1/zz_generated.deepcopy.go
  • machineconfiguration/v1alpha1/zz_generated.featuregated-crd-manifests.yaml
  • machineconfiguration/v1alpha1/zz_generated.swagger_doc_generated.go
  • payload-manifests/featuregates/featureGate-4-10-Hypershift-Default.yaml
  • payload-manifests/featuregates/featureGate-4-10-Hypershift-DevPreviewNoUpgrade.yaml
  • payload-manifests/featuregates/featureGate-4-10-Hypershift-OKD.yaml
  • payload-manifests/featuregates/featureGate-4-10-Hypershift-TechPreviewNoUpgrade.yaml
  • payload-manifests/featuregates/featureGate-4-10-SelfManagedHA-Default.yaml
  • payload-manifests/featuregates/featureGate-4-10-SelfManagedHA-DevPreviewNoUpgrade.yaml
  • payload-manifests/featuregates/featureGate-4-10-SelfManagedHA-OKD.yaml
  • payload-manifests/featuregates/featureGate-4-10-SelfManagedHA-TechPreviewNoUpgrade.yaml

📝 Walkthrough

Walkthrough

This pull request introduces two new OpenShift feature gates: MachineConfigNodes and PinnedImages. It adds corresponding Kubernetes Custom Resource Definitions (CRDs) in the v1alpha1 API group, including MachineConfigNode, MachineConfigNodeList, PinnedImageSet, and PinnedImageSetList types. Feature gate annotations are applied to various fields in v1 and v1alpha1 types. The changes include generated deepcopy methods, Swagger documentation, feature-gated CRD manifests, and test fixtures. Feature gate enablement is configured across multiple deployment profiles including Hypershift and SelfManagedHA environments.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
📝 Coding Plan
  • Generate coding plan for human review comments

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.11.3)

Error: build linters: unable to load custom analyzer "kubeapilinter": tools/_output/bin/kube-api-linter.so, plugin: not implemented
The command is terminated due to an error: build linters: unable to load custom analyzer "kubeapilinter": tools/_output/bin/kube-api-linter.so, plugin: not implemented


Comment @coderabbitai help to get the list of available commands and usage tips.

Tip

You can make CodeRabbit's review stricter and more nitpicky using the `assertive` profile, if that's what you prefer.

Change the reviews.profile setting to assertive to make CodeRabbit's nitpick more issues in your PRs.

@smg247
Copy link
Copy Markdown
Member Author

smg247 commented Mar 17, 2026

/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn-upgrade 10

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@smg247: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/4dfd2dc0-21f8-11f1-9cf7-049dfc073143-0

@smg247 smg247 changed the title Revert "OCPBUGS-74513: OCPBUGS-74519: OCPBUGS-78487: Remove MachineConfigNodes and PinnedImages feature gates and v1alpha1 references" TRT-2578: Revert "OCPBUGS-74513: OCPBUGS-74519: OCPBUGS-78487: Remove MachineConfigNodes and PinnedImages feature gates and v1alpha1 references" Mar 17, 2026
@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Mar 17, 2026
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Mar 17, 2026

@smg247: This pull request references TRT-2578 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This reverts #2715

Why the revert?

This PR is causing blocking job failures (aggregated-aws-ovn-upgrade, aggregated-azure-ovn-upgrade, aggregated-gcp-ovn-upgrade) in the ci amd64 payload 4.22.0-0.ci-2026-03-17-030533.

Evidence

  • Removes MachineConfigNodes and PinnedImages feature gates. Azure upgrade job shows 50% failure rate with MCO upgrade failures, cluster readiness failures, and node readiness issues. Feature gate removal on 4.21->4.22 upgrade path is a high-risk change. All 3 aggregated jobs started failing in this payload. Strong temporal and component match.

Tracking

  • Revert JIRA: TRT-2578
  • Original PR JIRA: OCPBUGS-74513, OCPBUGS-74519, OCPBUGS-78487

To unrevert

/cc @isabella-janssen

To restore your change, please:

  1. Revert this revert PR
  2. Layer a fix commit on top addressing the CI failures
  3. Ensure the following jobs pass before merging:
  • aggregated-aws-ovn-upgrade-4.22-minor-release-openshift-release-analysis-aggregator
  • aggregated-azure-ovn-upgrade-4.22-minor-release-openshift-release-analysis-aggregator
  • aggregated-gcp-ovn-upgrade-4.22-micro-release-openshift-release-analysis-aggregator

🤖 Generated with Revertomatic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@smg247
Copy link
Copy Markdown
Member Author

smg247 commented Mar 17, 2026

/hold for payload job verification

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 17, 2026
@smg247
Copy link
Copy Markdown
Member Author

smg247 commented Mar 17, 2026

/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-azure-ovn-upgrade 10

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@smg247: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/75fc36c0-21fa-11f1-9262-6f83fb5cd30a-0

@smg247
Copy link
Copy Markdown
Member Author

smg247 commented Mar 17, 2026

/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn-upgrade 10

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@smg247: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c99a28a0-21fa-11f1-8a96-867fbc82dd8b-0

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@smg247: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/lint b3678e8 link true /test lint
ci/prow/verify-feature-promotion b3678e8 link true /test verify-feature-promotion

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@smg247
Copy link
Copy Markdown
Member Author

smg247 commented Mar 17, 2026

/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn-upgrade 10

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@smg247: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-e2e-aws-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/2f52ae30-2203-11f1-803a-98d0f07118c2-0

@smg247
Copy link
Copy Markdown
Member Author

smg247 commented Mar 17, 2026

/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-azure-ovn-upgrade 10

@smg247
Copy link
Copy Markdown
Member Author

smg247 commented Mar 17, 2026

/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn-upgrade 10

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@smg247: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a559fb60-2203-11f1-838a-9bd172f7a6f5-0

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@smg247: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a14361b0-2203-11f1-8840-e041823f2467-0

@stbenjam
Copy link
Copy Markdown
Member

/pipeline required

@openshift-ci-robot
Copy link
Copy Markdown

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn
/test e2e-aws-ovn-hypershift
/test e2e-aws-ovn-hypershift-conformance
/test e2e-aws-ovn-techpreview
/test e2e-aws-serial-1of2
/test e2e-aws-serial-2of2
/test e2e-aws-serial-techpreview-1of2
/test e2e-aws-serial-techpreview-2of2
/test e2e-azure
/test e2e-gcp
/test e2e-upgrade
/test e2e-upgrade-out-of-change
/test minor-e2e-upgrade-minor

@stbenjam
Copy link
Copy Markdown
Member

/payload-aggregate periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade 5

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@stbenjam: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/7229e980-2230-11f1-9248-3bce635088b4-0

@stbenjam
Copy link
Copy Markdown
Member

We're not super confident about this revert, we're still investigating. It is not a permafail, in the originating payload it only showed up a few times, however these errors have never occurred, according to Sippy's 3 month history.

I also know kubelet service file is somehow tied in with which FG's are enabled, but unfortunately from what I can tell we sanitize this in our CI must gathers.

That all said...

I'm going to tag this so its ready to go in case we need it

Lint is erroneous because this was existing code (and lint checks PR diff only in this repo)
Verify feature gates also not quired here

/hold
/override ci/prow/lint
/override ci/prow/verify-feature-promotion
/lgtm
/label approved

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 17, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

@stbenjam: Overrode contexts on behalf of stbenjam: ci/prow/lint, ci/prow/verify-feature-promotion

Details

In response to this:

We're not super confident about this revert, we're still investigating. It is not a permafail, in the originating payload it only showed up a few times, however these errors have never occurred, according to Sippy's 3 month history.

I also know kubelet service file is somehow tied in with which FG's are enabled, but unfortunately from what I can tell we sanitize this in our CI must gathers.

That all said...

I'm going to tag this so its ready to go in case we need it

Lint is erroneous because this was existing code (and lint checks PR diff only in this repo)
Verify feature gates also not quired here

/hold
/override ci/prow/lint
/override ci/prow/verify-feature-promotion
/lgtm
/label approved

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Mar 17, 2026
@openshift-ci-robot
Copy link
Copy Markdown

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Mar 17, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: stbenjam
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 17, 2026
@stbenjam
Copy link
Copy Markdown
Member

@stbenjam stbenjam closed this Mar 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants