Skip to content

Conversation

@rrasouli
Copy link
Contributor

@rrasouli rrasouli commented Nov 17, 2025

Fix debug-winc-* jobs by adding missing tests-private base image

Problem

The debug-winc-* jobs for openshift-tests-private were failing with ImagePullBackOff errors:

Failed to pull image "image-registry.openshift-image-registry.svc:5000/ci-op-mgy0yhxc/stable:tests-private":
unable to pull image or OCI artifact: reading manifest tests-private in
image-registry.openshift-image-registry.svc:5000/ci-op-mgy0yhxc/stable: manifest unknown

Root Cause

The debug-winc-* jobs use the openshift-extended-test step which requires the stable:tests-private image.

For releases 4.20-4.21: The tests-private base image was not defined in the base_images section, and these versions don't build the image themselves.

For releases 4.16-4.19: These versions already BUILD the tests-private image internally (in the images section), so they don't need it as a base_image.

Solution

Added the missing tests-private base image configuration to all affected version configs:

tests-private:
  name: tests-private
  namespace: ci
  tag: "4.XX"  # version-specific (4.16, 4.17, 4.18, 4.19, 4.20, 4.21)

Changes Applied

✅ Versions Fixed (4.20-4.21)

  • release-4.20: Added tests-private base image
  • release-4.21: Added tests-private base image

ℹ️ Versions NOT Changed (4.16-4.19)

  • release-4.16: No change needed - builds tests-private image internally
  • release-4.17: No change needed - builds tests-private image internally
  • release-4.18: No change needed - builds tests-private image internally
  • release-4.19: No change needed - builds tests-private image internally

Impact Analysis

Affected Jobs (Fixed for 4.20-4.21)

Debug-winc-* jobs that use openshift-extended-test step:

  • release-4.20: debug-winc-{aws, gcp, azure, vsphere}-ipi (4 jobs)
  • release-4.21: debug-winc-{aws, gcp, azure, vsphere}-ipi (4 jobs)

Total: 8 debug-winc jobs fixed across 2 releases

Note on 4.16-4.19

Versions 4.16-4.19 build their own tests-private image (defined in the images section), so they reference stable:tests-private from their own build output, not from an external base_image.

Unaffected Jobs

Other jobs continue working as before:

  • title-check, bindata-check, code-check
  • e2e-aws, e2e-console (use different base images: tests-private-pr, tests-private-ui)
  • debug-disasterrecovery-* (use different test chains)

Why This Matters

Debug jobs are critical - they run in CD systems on every commit for Windows Container testing. The ImagePullBackOff error was completely blocking the Windows Container CI/CD pipeline.

Verification

Before Fix:

  • ❌ Jobs failed immediately with ImagePullBackOff: manifest tests-private unknown
  • ❌ Tests could not run at all due to missing image

After Fix:

  • tests-private image pulls successfully
  • ✅ Tests execute as expected
  • ✅ Verified on 4.21 via /pj-rehearse - image pull succeeds, tests run

Configuration Change

Before Fix

base_images:
  ci_tests-private-base:
    name: tests-private-base
    namespace: ci
    tag: latest
  cli:
    name: "4.XX"
    namespace: ocp
    tag: cli
  # tests-private: MISSING! ❌
  upi-installer:
    name: "4.XX"
    namespace: ocp
    tag: upi-installer

After Fix

base_images:
  ci_tests-private-base:
    name: tests-private-base
    namespace: ci
    tag: latest
  cli:
    name: "4.XX"
    namespace: ocp
    tag: cli
  tests-private:           # ✅ ADDED
    name: tests-private
    namespace: ci
    tag: "4.XX"
  upi-installer:
    name: "4.XX"
    namespace: ocp
    tag: upi-installer

References


Status: Ready for review. Versions 4.20-4.21 fixed by adding tests-private base image.

@openshift-ci openshift-ci bot requested review from juzhao and xingxingxia November 17, 2025 07:43
@rrasouli rrasouli marked this pull request as draft November 17, 2025 07:45
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 17, 2025
@rrasouli rrasouli changed the title Fix debug-winc-* jobs by adding tests-private base image for 4.21 Fix debug-winc-* jobs by adding tests-private base image Nov 17, 2025
@rrasouli
Copy link
Contributor Author

/retest

@openshift-ci openshift-ci bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 17, 2025
@rrasouli
Copy link
Contributor Author

/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-azure-ipi
/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-gcp-ipi
/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi
/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.17-debug-winc-nutanix-ipi

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot
Copy link
Contributor

@rrasouli: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command.

2 similar comments
@openshift-ci-robot
Copy link
Contributor

@rrasouli: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command.

@openshift-ci-robot
Copy link
Contributor

@rrasouli: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command.

@rrasouli
Copy link
Contributor Author

/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi pull-ci-openshift-openshift-tests-private-release-4.17-debug-winc-nutanix-ipi pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-aws-ipi

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot
Copy link
Contributor

@rrasouli: job(s): pull-ci-openshift-openshift-tests-private-release-4.17-debug-winc-nutanix-ipi either don't exist or were not found to be affected, and cannot be rehearsed

@rrasouli rrasouli force-pushed the fix-winc-debug-tests branch from 2e5a7b3 to 90e78f7 Compare November 17, 2025 14:05
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 17, 2025
@rrasouli rrasouli marked this pull request as ready for review November 17, 2025 14:07
@openshift-ci-robot
Copy link
Contributor

@rrasouli, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: invalid configuration: images[3]: duplicate image name 'tests-private' (previously defined by field 'base_images[tests-private]')
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 17, 2025
@rrasouli
Copy link
Contributor Author

/retest

The debug-winc-* jobs for releases 4.20-4.21 were failing with
ImagePullBackOff error because they use the openshift-extended-test
step which requires stable:tests-private image, but the tests-private
base image was not defined in the configuration.

This change adds the missing tests-private base image configuration
for releases 4.20 and 4.21:
- name: tests-private
- namespace: ci
- tag: "4.20" / "4.21"

Note: Releases 4.16-4.19 do not need this change as they build the
tests-private image internally in their images section.

This fix affects the debug-winc-* jobs (aws, gcp, azure, vsphere)
that use openshift-extended-test step for releases 4.20-4.21.
@rrasouli rrasouli force-pushed the fix-winc-debug-tests branch from 90e78f7 to 84e6227 Compare November 17, 2025 17:38
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@rrasouli: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-openshift-tests-private-release-4.21-bindata-check openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-code-check openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-disasterrecovery-aws-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-disasterrecovery-baremetal-upi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-aws-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-azure-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-gcp-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-e2e-aws openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-e2e-console openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-images openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-images-images openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.21-title-check openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-bindata-check openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-code-check openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-disasterrecovery-aws-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-disasterrecovery-baremetal-upi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-aws-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-azure-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-gcp-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-e2e-aws openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-e2e-console openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-images openshift/openshift-tests-private presubmit Ci-operator config changed
pull-ci-openshift-openshift-tests-private-release-4.20-images-images openshift/openshift-tests-private presubmit Ci-operator config changed

A total of 1958 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@rrasouli
Copy link
Contributor Author

/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-aws-ipi pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-azure-ipi

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@rrasouli
Copy link
Contributor Author

/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-azure-ipi

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@rrasouli
Copy link
Contributor Author

/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-vsphere-ipi

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 17, 2025

@rrasouli: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/openshift/openshift-tests-private/release-4.21/debug-winc-aws-ipi 84e6227 link unknown /pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-aws-ipi
ci/rehearse/openshift/openshift-tests-private/release-4.21/debug-winc-vsphere-ipi 84e6227 link unknown /pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi
ci/rehearse/openshift/openshift-tests-private/release-4.20/debug-winc-gcp-ipi 84e6227 link unknown /pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-gcp-ipi
ci/rehearse/openshift/openshift-tests-private/release-4.20/debug-winc-azure-ipi 84e6227 link unknown /pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-azure-ipi
ci/rehearse/openshift/openshift-tests-private/release-4.21/debug-winc-gcp-ipi 84e6227 link unknown /pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-gcp-ipi
ci/rehearse/openshift/openshift-tests-private/release-4.21/debug-winc-azure-ipi 84e6227 link unknown /pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-azure-ipi
ci/rehearse/openshift/openshift-tests-private/release-4.20/debug-winc-vsphere-ipi 84e6227 link unknown /pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-vsphere-ipi

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 18, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liangxia, rrasouli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rrasouli
Copy link
Contributor Author

/pj-rehearse ack

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Nov 19, 2025
@openshift-merge-bot openshift-merge-bot bot merged commit fc347b3 into openshift:master Nov 19, 2025
13 of 20 checks passed
rrasouli added a commit to rrasouli/release that referenced this pull request Nov 20, 2025
openshift#71432)

The debug-winc-* jobs for releases 4.20-4.21 were failing with
ImagePullBackOff error because they use the openshift-extended-test
step which requires stable:tests-private image, but the tests-private
base image was not defined in the configuration.

This change adds the missing tests-private base image configuration
for releases 4.20 and 4.21:
- name: tests-private
- namespace: ci
- tag: "4.20" / "4.21"

Note: Releases 4.16-4.19 do not need this change as they build the
tests-private image internally in their images section.

This fix affects the debug-winc-* jobs (aws, gcp, azure, vsphere)
that use openshift-extended-test step for releases 4.20-4.21.
dfrazzette pushed a commit to dfrazzette/release that referenced this pull request Dec 9, 2025
openshift#71432)

The debug-winc-* jobs for releases 4.20-4.21 were failing with
ImagePullBackOff error because they use the openshift-extended-test
step which requires stable:tests-private image, but the tests-private
base image was not defined in the configuration.

This change adds the missing tests-private base image configuration
for releases 4.20 and 4.21:
- name: tests-private
- namespace: ci
- tag: "4.20" / "4.21"

Note: Releases 4.16-4.19 do not need this change as they build the
tests-private image internally in their images section.

This fix affects the debug-winc-* jobs (aws, gcp, azure, vsphere)
that use openshift-extended-test step for releases 4.20-4.21.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants