Add Windows BYOH provisioning support to Prow CI #73785

rrasouli · 2026-01-21T11:50:27Z

Add Windows BYOH Provisioning Support to Prow CI

This PR adds step-registry components for provisioning Windows nodes via BYOH (Bring Your Own Host) using https://github.com/openshift/terraform-windows-provisioner.

Overview

BYOH is the default Windows node provisioning method for Windows Container testing in OpenShift. This PR integrates BYOH provisioning into the Prow CI step-registry, enabling automated Windows node deployment across all supported platforms.

Changes

New Step-Registry Components

1. Windows BYOH Steps (windows/byoh/)

windows-byoh-provision: Provisions Windows nodes using terraform-windows-provisioner
- Supports AWS, Azure, GCP, vSphere, Nutanix, and platform "none"
- Auto-configures credentials from cluster profile secrets
- SSH key extraction: Loads directly from ${CLUSTER_PROFILE_DIR}/ssh-publickey
- Unique instance naming: Random 3-digit suffix (000-999) to avoid conflicts from previous failed runs
- Terraform state persistence: Uses ${ARTIFACT_DIR}/terraform_byoh/ (SHARED_DIR is cluster-profile mount, not shared between steps)
- Exports instance info in WMCO BYOH format for e2e tests
windows-byoh-destroy: Cleanup step for BYOH Windows nodes
- Passes all required arguments to match provision step signature
- SSH key extraction: Loads from cluster profile (required by byoh.sh)
- Reads instance name from ARTIFACT_DIR (saved by provision step)
- Runs terraform destroy using state from ${ARTIFACT_DIR}/terraform_byoh/
- No fallback mechanisms: Fails if terraform state not found (state is always available with ARTIFACT_DIR fix)
- Clean, predictable destroy process using only Terraform

2. Deprovision Chains (All Platforms)

Created deprovision chains with proper cleanup ordering:

cucushift-installer-rehearse-aws-ipi-ovn-winc-deprovision
cucushift-installer-rehearse-azure-ipi-ovn-winc-deprovision
cucushift-installer-rehearse-gcp-ipi-ovn-winc-deprovision
cucushift-installer-rehearse-vsphere-ipi-ovn-winc-deprovision
cucushift-installer-rehearse-nutanix-ipi-ovn-winc-deprovision
cucushift-installer-rehearse-aws-upi-ovn-winc-deprovision

Cleanup Order:

Gather diagnostics (gather, gather-aws-console) - Collects logs while nodes exist
Destroy BYOH nodes (windows-byoh-destroy) - Cleans up Windows instances with proper arguments
Platform deprovision - Tears down cluster infrastructure

3. Updated Provision Chains (All Platforms)

Added windows-byoh-provision as mandatory step to all winc chains:

cucushift-installer-rehearse-aws-ipi-ovn-winc-provision
cucushift-installer-rehearse-azure-ipi-ovn-winc-provision
cucushift-installer-rehearse-gcp-ipi-ovn-winc-provision
cucushift-installer-rehearse-vsphere-ipi-ovn-winc-provision
cucushift-installer-rehearse-nutanix-ipi-ovn-winc-provision

4. Updated Workflows (All Platforms)

Updated workflows to use new deprovision chains in post: section:

cucushift-installer-rehearse-aws-ipi-ovn-winc-workflow
cucushift-installer-rehearse-azure-ipi-ovn-winc-workflow
cucushift-installer-rehearse-gcp-ipi-ovn-winc-workflow
cucushift-installer-rehearse-vsphere-ipi-ovn-winc-workflow
cucushift-installer-rehearse-nutanix-ipi-ovn-winc-workflow

5. New Platform "None" Chain and Workflow

cucushift-installer-rehearse-aws-upi-ovn-winc-provision
cucushift-installer-rehearse-aws-upi-ovn-winc-deprovision
cucushift-installer-rehearse-aws-upi-ovn-winc-workflow
- UPI (platform=none) on AWS with Windows BYOH
- BYOH-only (no MachineSets support on platform "none")

Platform Coverage

Platform	Deployment Method	Chain Type	Status
AWS IPI	MachineSet + BYOH	IPI + BYOH	✅
Azure IPI	MachineSet + BYOH	IPI + BYOH	✅
GCP IPI	MachineSet + BYOH	IPI + BYOH	✅
vSphere IPI	MachineSet + BYOH	IPI + BYOH	✅
Nutanix IPI	MachineSet + BYOH	IPI + BYOH	✅
AWS UPI (platform "none")	BYOH only	BYOH only	✅

How It Works

Provision Step Flow

Use Pre-built Image: Runs in terraform-windows-provisioner container image with all dependencies
Generate Unique Name: Random 3-digit suffix (e.g., byoh-586, byoh-123) to avoid conflicts
Save Instance Name: Writes instance name to ${ARTIFACT_DIR}/byoh_instance_name.txt for destroy step
Extract SSH Key: Loads from ${CLUSTER_PROFILE_DIR}/ssh-publickey (same key used by WMCO)
Detect Platform: Auto-detects platform from cluster infrastructure
Load Credentials: Cloud credentials from CLUSTER_PROFILE_DIR (auto-provided by Prow)
Provision Nodes: Runs terraform-windows-provisioner via byoh.sh apply <name> <num> "" <version>
Terraform State: Saved to ${ARTIFACT_DIR}/terraform_byoh/<platform>/ (persistent shared volume)
Wait for Ready: Waits up to 30 minutes for Windows nodes to become Ready
Export Instance Info: Creates files in ${SHARED_DIR}/<ip>_windows_instance.txt format for WMCO BYOH e2e tests

Destroy Step Flow

Use Pre-built Image: Runs in terraform-windows-provisioner container image
Read Instance Name: Loads from ${ARTIFACT_DIR}/byoh_instance_name.txt (saved by provision)
Extract SSH Key: Loads from ${CLUSTER_PROFILE_DIR}/ssh-publickey (required by byoh.sh)
Detect Platform: Auto-detects platform from cluster infrastructure
Access Terraform State: State is in ${ARTIFACT_DIR}/terraform_byoh/<platform>/ (persistent shared volume)
Verify State Exists: Fails immediately if terraform state not found (ensures proper cleanup)
Run Terraform Destroy: Executes byoh.sh destroy <name> <num> "" <version> with all arguments matching provision

Note: No cloud CLI fallback - terraform state is guaranteed to exist with ARTIFACT_DIR fix

Integration with WMCO Tests

The instance info format is compatible with existing WMCO BYOH e2e tests:

# Exported by windows-byoh-provision
${SHARED_DIR}/10.0.0.100_windows_instance.txt:
  username: Administrator  # or 'capi' for Azure

Consumed by windows-e2e-operator-test-byoh

Configuration

Default Values

All values are overridable via environment variables:

BYOH_INSTANCE_NAME="byoh-<XXX>"      # Auto-generated: random 3-digit suffix (000-999)
                                      # Example: byoh-586, byoh-123
                                      # Ensures unique names to avoid conflicts
BYOH_NUM_WORKERS="2"                 # Number of Windows nodes
BYOH_WINDOWS_VERSION="2022"          # Windows Server version (2019|2022)
WINC_ADMIN_PASSWORD="<auto-gen>"     # Auto-generated if not set
BYOH_READY_TIMEOUT="30m"             # Timeout for nodes to become Ready (matches production)

Platform-Specific Settings

All platform-specific configuration (instance types, disk sizes, network settings) is auto-detected from the cluster or can be overridden via environment variables. See https://github.com/openshift/terraform-windows-provisioner for details.

Review Comments Addressed

All feedback from PR #71002 has been addressed:

@jianlinliu Comments

✅ Chicken-and-egg problem: Fixed via 2-PR approach
- PR Bootstrap CI configuration for terraform-windows-provisioner #73680 (merged): Bootstrap image build configuration
- Image now available: registry.ci.openshift.org/ci/terraform-windows-provisioner:latest
✅ No terraform downloads: Terraform 1.9.5 pre-installed in Dockerfile
✅ No git clone: Scripts pre-copied to /usr/local/share/byoh-provisioner/ in image
✅ State in ${SHARED_DIR}: CRITICAL FIX - SHARED_DIR was wrong!
- Problem: SHARED_DIR=/tmp/secret points to cluster-profile secret mount (container-local, NOT shared between steps)
- Solution: Use ARTIFACT_DIR=/logs/artifacts (actual persistent shared volume)
- Terraform state now correctly stored at ${ARTIFACT_DIR}/terraform_byoh/<platform>/
- State persists from provision to destroy step
✅ Destroy not working: Fixed by:
- Exporting BYOH_WINDOWS_VERSION in destroy step
- Passing all 4 arguments to byoh.sh destroy to match provision signature
- Extracting SSH public key (required by byoh.sh even for destroy)
- Using ARTIFACT_DIR for state persistence (SHARED_DIR was the root cause)
- Reading instance name from ${ARTIFACT_DIR}/byoh_instance_name.txt
- Removed cloud CLI fallback (no longer needed with ARTIFACT_DIR fix)
✅ Unique Instance Naming: Prevents conflicts from previous failed runs
- Uses random 3-digit suffix (000-999): byoh-586, byoh-123, etc.
- Respects Azure's 15-char VM name limit
- Instance name saved to ${ARTIFACT_DIR}/byoh_instance_name.txt for destroy step
- Both provision and destroy read from same file

@jrvaldes Comments

✅ OWNERS file locations: Fixed by using symlinks for proper ownership inheritance

Key Debugging & Fixes

Critical Issue: Secret Size Limit (3MB)

Problem Discovered: Provision succeeded but Secret update failed:

error: failed to create/update secret: Request entity too large: limit is 3145728

Root Cause: SHARED_DIR is backed by a Kubernetes Secret with a 3MB size limit. Initial approach tarred the entire terraform_byoh/ directory which includes:

terraform.tfstate (~50-100KB)
.terraform/ provider binaries and cache (several MB)
Total size exceeded Secret limit

Investigation:

SHARED_DIR files are shared between steps, but subfolders are NOT
Cannot use subdirectories like ${SHARED_DIR}/terraform_byoh/${PLATFORM}/
Tarring entire directory exceeded 3MB limit
Only the terraform.tfstate file is needed for destroy

Solution: Save only terraform.tfstate file to SHARED_DIR

Provision: cp terraform.tfstate → ${SHARED_DIR}/terraform_byoh_${PLATFORM}.tfstate
Destroy: cp back to ${ARTIFACT_DIR}/terraform_byoh/${PLATFORM}/terraform.tfstate
.terraform/ stays in ARTIFACT_DIR (preserved for debugging, not shared)

Impact: This was causing:

Complete failure to save ANY files to SHARED_DIR (Secret update rejected)
Destroy step found nothing (no marker files, no instance names, no terraform state)
Orphaned BYOH instances (destroy couldn't run)
Failed deprovision steps

Other Fixes

Random Suffix: Changed from PROW_JOB_ID (not available) to random 3-digit number
Wait Timeout: Reduced from 45m to 30m (matches flexy-templates production)
Removed CLI Fallback: With ARTIFACT_DIR fix, terraform state always available - no need for gcloud/aws/az fallback
Hardcoded Defaults: Removed BYOH_INSTANCE_NAME="byoh-winc" from ref YAML (was overriding random suffix)

Testing

Bootstrap PR Status

✅ PR Bootstrap CI configuration for terraform-windows-provisioner #73680 merged: Image build configuration
✅ Image available: registry.ci.openshift.org/ci/terraform-windows-provisioner:latest
✅ Image digest: sha256:a215c67c2d041d2bb6ee3640fed33d6e9ca9cb1a52c540a2be982aeb21904611

Rehearsal Results

Validated with rehearsals on AWS IPI 4.17:

✅ Provision step succeeded: 16m17s runtime
- SSH key extraction working
- Both MachineSet and BYOH nodes provisioned
- All Windows nodes became Ready
✅ Destroy fixes validated: All arguments now passed correctly

Dependencies

This step-registry uses terraform-windows-provisioner container image for provisioning Windows nodes:

Repository: terraform-windows-provisioner
Container Image: registry.ci.openshift.org/ci/terraform-windows-provisioner:latest
Build Config: ci-operator/config/openshift/terraform-windows-provisioner/
Status: Production-ready, vendor-neutral tool

Container Image Includes:

Terraform 1.9.5 - Pre-installed, no download needed
Provisioner Scripts - byoh.sh and platform-specific configs
Dependencies - oc, kubectl, jq, bash

Benefits of Pre-built Image:

No rate limits - Avoids terraform download and git clone rate limits during parallel job execution
Faster execution - Pull cached image vs downloading dependencies
Consistency - Same terraform version across all jobs
Reliability - Pre-validated image with all dependencies

Migration Path

This PR adds BYOH as a mandatory step to all winc chains. Existing Windows testing workflows will now provision nodes via both MachineSets (IPI platforms) and BYOH.

Future Work

Planned for separate PRs:

Create actual periodic/presubmit job definitions using these chains
Add upgrade testing workflows (N+1 scenarios)
Complete migration from Jenkins Flexy to Prow CI for Windows testing

Files Changed

ci-operator/step-registry/
├── windows/byoh/
│   ├── OWNERS (symlink)
│   ├── provision/
│   │   ├── OWNERS (symlink)
│   │   ├── windows-byoh-provision-commands.sh
│   │   ├── windows-byoh-provision-ref.yaml
│   │   └── windows-byoh-provision-ref.metadata.json
│   └── destroy/
│       ├── OWNERS (symlink)
│       ├── windows-byoh-destroy-commands.sh
│       ├── windows-byoh-destroy-ref.yaml
│       └── windows-byoh-destroy-ref.metadata.json
├── cucushift/installer/rehearse/
│   ├── aws/
│   │   ├── ipi/ovn/winc/
│   │   │   ├── provision/ (updated)
│   │   │   └── deprovision/ (new)
│   │   └── upi/ovn/winc/ (new - platform "none")
│   │       ├── provision/
│   │       └── deprovision/
│   ├── azure/ipi/ovn/winc/
│   │   ├── provision/ (updated)
│   │   └── deprovision/ (new)
│   ├── gcp/ipi/ovn/winc/
│   │   ├── provision/ (updated)
│   │   └── deprovision/ (new)
│   ├── vsphere/ipi/ovn/winc/
│   │   ├── provision/ (updated)
│   │   └── deprovision/ (new)
│   └── nutanix/ipi/ovn/winc/
│       ├── provision/ (updated)
│       └── deprovision/ (new)

Replaces: PR Add Windows BYOH provisioning support to Prow CI #71002 (closed due to rebase force-push)
Bootstrap PR: Bootstrap CI configuration for terraform-windows-provisioner #73680 (merged)
Companion PR: Fix debug-winc jobs by adding tests-private base image for 4.20-4.23 #73778 (merged) - Fixes debug-winc jobs
Jira: WINC-1473 - Windows Container upgrade testing requirements
Upstream Project: terraform-windows-provisioner

/assign @jianlinliu
/cc @jrvaldes @weinliu

rrasouli · 2026-01-21T12:08:10Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-main-debug-vshphere-gcp-ipi

openshift-ci-robot · 2026-01-21T12:08:13Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci-robot · 2026-01-21T12:10:36Z

@rrasouli: job(s): pull-ci-openshift-openshift-tests-private-main-debug-vshphere-gcp-ipi either don't exist or were not found to be affected, and cannot be rehearsed

openshift-ci · 2026-01-21T17:14:13Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rrasouli
Once this PR has been reviewed and has the lgtm label, please ask for approval from jianlinliu and additionally assign sebsoto for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rrasouli · 2026-01-21T17:15:13Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-vspere-ipi

openshift-ci-robot · 2026-01-21T17:15:16Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci-robot · 2026-01-21T17:18:02Z

@rrasouli: job(s): pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-vspere-ipi either don't exist or were not found to be affected, and cannot be rehearsed

rrasouli · 2026-01-22T05:09:50Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi eriodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-22T05:09:53Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci-robot · 2026-01-22T05:12:41Z

@rrasouli: job(s): eriodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14 either don't exist or were not found to be affected, and cannot be rehearsed

This adds step-registry components and chains for provisioning Windows nodes via BYOH (Bring Your Own Host) using terraform-windows-provisioner. Changes: - Add windows-byoh-provision step (uses terraform-windows-provisioner) - Add windows-byoh-destroy step with cloud CLI fallback mechanism - Update all winc chains to include BYOH provisioning as mandatory step - Add deprovision chains to destroy BYOH nodes before cluster teardown - Add platform "none" (UPI AWS) chain for Windows BYOH - Export instance info in WMCO BYOH format for e2e tests - Use upi-installer image for cloud CLI access (aws, gcloud, az) Platform coverage: - AWS IPI (MachineSet + BYOH) - Azure IPI (MachineSet + BYOH) - GCP IPI (MachineSet + BYOH) - vSphere IPI (MachineSet + BYOH) - Nutanix IPI (MachineSet + BYOH) - AWS UPI platform "none" (BYOH only) Destroy fallback mechanism: When Terraform state is unavailable, the destroy step uses cloud CLI to find and delete instances by name pattern, preventing orphaned instances that could block cluster network deletion. BYOH is now the default Windows node provisioning method for all Windows Container CI testing.

rrasouli · 2026-01-22T08:19:55Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi eriodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-22T08:19:58Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-01-22T08:21:34Z

/test all

openshift-ci-robot · 2026-01-22T08:22:46Z

@rrasouli: job(s): eriodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14 either don't exist or were not found to be affected, and cannot be rehearsed

rrasouli · 2026-01-22T12:42:40Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-22T12:42:43Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

- Generate random 3-digit suffix for unique instance names (byoh-XXX) - Save instance name to SHARED_DIR for destroy step to read - Fix wait for Ready nodes logic (exclude NotReady status) - Reduce timeout to 30 minutes (matches flexy-templates production) - Fix destroy script indentation for shellcheck compliance - Add automatic fallback to default name if shared file missing

rrasouli · 2026-01-22T16:38:50Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-23T10:18:40Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-01-23T12:15:00Z

/test step-registry-shellcheck

rrasouli · 2026-01-23T12:23:23Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-23T12:23:27Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-01-23T16:30:09Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-23T16:30:12Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-01-24T04:17:21Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi eriodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-24T04:17:24Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci-robot · 2026-01-24T04:20:03Z

@rrasouli: job(s): eriodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14 either don't exist or were not found to be affected, and cannot be rehearsed

rrasouli · 2026-01-24T04:27:19Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-aws-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi eriodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-24T04:27:21Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci-robot · 2026-01-24T04:29:58Z

@rrasouli: job(s): eriodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14 either don't exist or were not found to be affected, and cannot be rehearsed

rrasouli · 2026-01-24T06:02:21Z

/test debug-winc-vsphere-ipi

openshift-ci · 2026-01-24T06:02:25Z

@rrasouli: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test app-ci-config-dry

/test boskos-config

/test boskos-config-generation

/test build01-dry

/test build02-dry

/test build03-dry

/test build04-dry

/test build06-dry

/test build07-dry

/test build08-dry

/test build09-dry

/test build10-dry

/test check-gh-automation

/test check-gh-automation-tide

/test ci-operator-config

/test ci-operator-config-metadata

/test ci-operator-registry

/test ci-secret-bootstrap-config-validation

/test ci-testgrid-allow-list

/test clusterimageset-validate

/test config

/test core-ci-config-dry

/test core-valid

/test generated-config

/test generated-dashboards

/test hosted-mgmt-dry

/test image-mirroring-config-validation

/test jira-lifecycle-config

/test labels

/test openshift-image-mirror-mappings

/test ordered-prow-config

/test owners

/test pr-reminder-config

/test prow-config

/test prow-config-filenames

/test prow-config-semantics

/test pylint

/test release-config

/test release-controller-config

/test rover-groups-config-validation

/test secret-generator-config-valid

/test services-valid

/test stackrox-stackrox-stackrox-stackrox-check

/test step-registry-metadata

/test step-registry-shellcheck

/test sync-rover-groups

/test verified-config

/test vsphere02-dry

/test yamllint

The following commands are available to trigger optional jobs:

/test build05-dry

/test build11-dry

/test check-cluster-profiles-config

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-release-check-gh-automation

pull-ci-openshift-release-master-ci-operator-config

pull-ci-openshift-release-master-ci-operator-config-metadata

pull-ci-openshift-release-master-ci-operator-registry

pull-ci-openshift-release-master-core-valid

pull-ci-openshift-release-master-generated-config

pull-ci-openshift-release-master-ordered-prow-config

pull-ci-openshift-release-master-owners

pull-ci-openshift-release-master-prow-config-filenames

pull-ci-openshift-release-master-release-controller-config

pull-ci-openshift-release-master-step-registry-metadata

pull-ci-openshift-release-master-step-registry-shellcheck

pull-ci-openshift-release-openshift-image-mirror-mappings

pull-ci-openshift-release-yamllint

Details

In response to this:

/test debug-winc-vsphere-ipi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

rrasouli · 2026-01-24T06:03:01Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14

openshift-ci-robot · 2026-01-24T06:03:03Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

- SHARED_DIR backed by K8s Secret with 3MB limit - Tarring entire terraform_byoh/ dir exceeded limit (includes .terraform/ providers) - Now only copy terraform.tfstate file (small, <100KB) - Provision: cp to SHARED_DIR/terraform_byoh_${PLATFORM}.tfstate - Destroy: cp back to ARTIFACT_DIR before terraform destroy Fixes: error: failed to create/update secret: Request entity too large: limit is 3145728

openshift-ci-robot · 2026-01-24T09:17:04Z

[REHEARSALNOTIFIER]
@rrasouli: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-openshift-tests-private-main-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.23-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.22-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.18-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.17-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.16-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.13-debug-winc-aws-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-main-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.23-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.22-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.18-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.17-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.16-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.13-debug-winc-azure-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.23-debug-winc-gcp-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.22-debug-winc-gcp-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-gcp-ipi	openshift/openshift-tests-private	presubmit	Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-gcp-ipi	openshift/openshift-tests-private	presubmit	Registry content changed

A total of 99 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

rrasouli · 2026-01-24T09:28:43Z

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14 pull-ci-openshift-openshift-tests-private-main-debug-winc-gcp-ipi

openshift-ci-robot · 2026-01-24T09:28:46Z

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

rrasouli · 2026-01-24T14:39:13Z

Closing in favor of clean PR #73920 with single squashed commit and simplified description.

openshift-ci · 2026-01-24T14:39:19Z

@rrasouli: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/openshift/openshift-tests-private/release-4.19/debug-winc-azure-ipi	`6dd9421`	link	unknown	`/pj-rehearse pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-azure-ipi`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot assigned jianlinliu Jan 21, 2026

openshift-ci bot requested review from anuragthehatter and gpei January 21, 2026 11:50

rrasouli force-pushed the add-windows-byoh-prow-ci branch from 5bb51ef to 21ecb46 Compare January 21, 2026 17:11

rrasouli force-pushed the add-windows-byoh-prow-ci branch from 21ecb46 to e91f31b Compare January 22, 2026 05:08

rrasouli force-pushed the add-windows-byoh-prow-ci branch from e91f31b to 4bbde3a Compare January 22, 2026 08:15

rrasouli marked this pull request as draft January 22, 2026 08:15

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 22, 2026

rrasouli marked this pull request as ready for review January 22, 2026 08:23

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 22, 2026

rrasouli force-pushed the add-windows-byoh-prow-ci branch from 902cde2 to 45c9276 Compare January 22, 2026 16:34

rrasouli force-pushed the add-windows-byoh-prow-ci branch from 7258872 to c335a5c Compare January 23, 2026 16:18

rrasouli force-pushed the add-windows-byoh-prow-ci branch from c335a5c to 6dd9421 Compare January 24, 2026 04:06

rrasouli marked this pull request as draft January 24, 2026 04:25

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 24, 2026

rrasouli marked this pull request as ready for review January 24, 2026 04:25

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 24, 2026

rrasouli force-pushed the add-windows-byoh-prow-ci branch from 6dd9421 to 8e73558 Compare January 24, 2026 09:14

rrasouli closed this Jan 24, 2026

Add Windows BYOH provisioning support to Prow CI #73785

Add Windows BYOH provisioning support to Prow CI #73785

Uh oh!

Conversation

rrasouli commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add Windows BYOH Provisioning Support to Prow CI

Overview

Changes

New Step-Registry Components

1. Windows BYOH Steps (windows/byoh/)

2. Deprovision Chains (All Platforms)

3. Updated Provision Chains (All Platforms)

4. Updated Workflows (All Platforms)

5. New Platform "None" Chain and Workflow

Platform Coverage

How It Works

Provision Step Flow

Destroy Step Flow

Integration with WMCO Tests

Configuration

Default Values

Platform-Specific Settings

Review Comments Addressed

@jianlinliu Comments

@jrvaldes Comments

Key Debugging & Fixes

Critical Issue: Secret Size Limit (3MB)

Other Fixes

Testing

Bootstrap PR Status

Rehearsal Results

Dependencies

Container Image Includes:

Benefits of Pre-built Image:

Migration Path

Future Work

Files Changed

Related

Uh oh!

rrasouli commented Jan 21, 2026

Uh oh!

openshift-ci-robot commented Jan 21, 2026

Uh oh!

openshift-ci-robot commented Jan 21, 2026

Uh oh!

openshift-ci bot commented Jan 21, 2026

Uh oh!

rrasouli commented Jan 21, 2026

Uh oh!

openshift-ci-robot commented Jan 21, 2026

Uh oh!

openshift-ci-robot commented Jan 21, 2026

Uh oh!

rrasouli commented Jan 22, 2026

Uh oh!

openshift-ci-robot commented Jan 22, 2026

Uh oh!

openshift-ci-robot commented Jan 22, 2026

Uh oh!

rrasouli commented Jan 22, 2026

Uh oh!

openshift-ci-robot commented Jan 22, 2026

Uh oh!

rrasouli commented Jan 22, 2026

Uh oh!

openshift-ci-robot commented Jan 22, 2026

Uh oh!

rrasouli commented Jan 22, 2026

Uh oh!

openshift-ci-robot commented Jan 22, 2026

Uh oh!

rrasouli commented Jan 22, 2026

Uh oh!

openshift-ci-robot commented Jan 23, 2026

Uh oh!

rrasouli commented Jan 23, 2026

Uh oh!

rrasouli commented Jan 23, 2026

Uh oh!

rrasouli commented Jan 21, 2026 •

edited

Loading