[4.10] Generate ci-operator configuration and jobs by petr-muller · Pull Request #19861 · openshift/release

petr-muller · 2021-06-30T17:59:33Z

[4.10] Generate ci-operator configuration
[4.10] Generate ci-operator configuration for openshift-priv
[4.10] Generate vanilla jobs for 4.10
[4.10] Carry job customization over from master/main jobs
[4.10]: Update template allowlist

Compared to the 4.9 PR, I
changed the script to first generate a vanilla version of 4.10 jobs, and then
carried over the customizations from master/main branches (instead of 4.9
branches where the modifications may be stale), creating a separate commit
just for this, so that custom modifications carried over to 4.10 can be inspected.

The carry-over should be repeated at the time of 4.9 code freeze for 4.9 etc, but
hopefully at that time we'll have the first-class support in ci-operator and the
carry-over will be done by config-brancher.

#!/bin/bash

set -o nounset
set -o errexit
set -o pipefail

git fetch upstream
git reset --hard upstream/master

CURRENT=4.9
NEXT=4.10

config-brancher --config-dir ci-operator/config --current-release=$CURRENT --future-release=$NEXT --confirm
git add ci-operator/config
git commit -m "[$NEXT] Generate ci-operator configuration"

ci-operator-config-mirror --config-path ci-operator/config --to-org openshift-priv
git add ci-operator/config
git commit -m "[$NEXT] Generate ci-operator configuration for openshift-priv"

make jobs
git add ci-operator/jobs
git commit -m "[$NEXT] Generate vanilla jobs for $NEXT"

cat <<EOF >/tmp/filter.py
#!/usr/bin/env python3

import yaml
import sys
import copy

with open(sys.argv[1]) as f:
    all = yaml.full_load(f)
    for t in ("presubmits", "postsubmits"):
        for repo in all.get(t, {}):
            pruned = []
            for job in all.get(t, {}).get(repo, []):
                if job.get("agent", "") == "kubernetes":
                    pruned.append(job)
            all[t][repo] = pruned

with open(sys.argv[1], 'w') as f:
    yaml.dump(all, f, default_flow_style=False)
EOF

chmod +x /tmp/filter.py

find ci-operator/jobs/ -name '*-release-'$CURRENT'-*submits.yaml' -or -name '*-'$CURRENT'-periodics.yaml' | while read -r item; do
    echo -n "Processing: ${item}"
    for main in master main; do
      if [ -e "${item/release-$CURRENT/$main}" ]; then
        cp "${item/release-$CURRENT/$main}" "${item/$CURRENT/$NEXT}"
        sed -i "s/-$main-/-release-$NEXT-/g" "${item/$CURRENT/$NEXT}" # job names
        sed -i "s/- $main/- release-$NEXT/g" "${item/$CURRENT/$NEXT}" # branches
        /tmp/filter.py "${item/$CURRENT/$NEXT}"
        break
      fi
    done
    echo " [DONE]"
done

# openshift/release are special and are handled separately
rm -f ci-operator/jobs/openshift/release/openshift-release-release-$NEXT-periodics.yaml
make jobs
git add ci-operator/jobs
git commit -m "[$NEXT] Carry job customization over from master/main jobs"


make template-allowlist
git add core-services/template-deprecation/_allowlist.yaml
git commit -m "[$NEXT] Update template allowlist"

/hold
Needs #19848

openshift-ci · 2021-06-30T17:59:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [petr-muller]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

petr-muller · 2021-06-30T18:20:00Z

/hold

LOL no 4.10 code yet

petr-muller · 2021-07-01T11:42:20Z

/test correctly-sharded-config

petr-muller · 2021-07-01T12:36:59Z

Needs openshift/ci-tools#2136 to fix ci/prow/config

petr-muller · 2021-08-03T14:30:59Z

/hold
Some of the rehearsals look fishy, need to investigate.

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/19861/rehearse-19861-pull-ci-openshift-cluster-api-actuator-pkg-release-4.10-images/1422552358713298944

ERRO[2021-08-03T13:38:12Z] Failed to load arguments.                     error=invalid configuration: you must define at least one test or image build in 'tests' or 'images'

petr-muller · 2021-08-03T18:18:34Z

Hit openshift/ci-tools#2217

petr-muller · 2021-08-03T18:18:45Z

/test pj-rehearse

openshift-ci · 2021-08-04T15:34:47Z

@petr-muller: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/rehearse/openshift/cluster-kube-descheduler-operator/release-4.10/images	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-nfs/release-4.10/vendor	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/compliance-operator/release-4.10/verify	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/compliance-operator/release-4.10/e2e-aws	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/etcd/openshift-4.10/e2e-aws-serial	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/etcd/openshift-4.10/configmap-scale	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-kube-descheduler-operator/release-4.10/unit	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-kube-descheduler-operator/release-4.10/e2e-aws-operator	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-nfs/release-4.10/e2e-openstack-csi	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-kube-descheduler-operator/release-4.10/verify	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-nfs/release-4.10/go-fmt	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-nfs/release-4.10/images	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-nfs/release-4.10/go-vet	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/compliance-operator/release-4.10/unit	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/compliance-operator/release-4.10/go-build	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/compliance-operator/release-4.10/images	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/etcd/openshift-4.10/e2e-aws	7bf002c5168adca70ade64cc36fc1a93d2c619fb	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-api-actuator-pkg/release-4.10/images	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-api-actuator-pkg/release-4.10/e2e-gcp-operator	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-api-actuator-pkg/release-4.10/e2e-vsphere-operator	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-api-actuator-pkg/release-4.10/e2e-azure-operator	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-api-actuator-pkg/release-4.10/e2e-aws-operator	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/openshift/cluster-api-actuator-pkg/release-4.10/e2e-aws-operator-tech-preview	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/openshift/aws-ebs-csi-driver-operator/release-4.10/e2e-aws-csi-migration	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/openshift/kubecsr/openshift-4.10/e2e-aws	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/openshift/aws-ebs-csi-driver-operator/release-4.10/e2e-aws-csi	8e6aa696e4528b1dff7ceae5d0eddeccd2bb3d86	link	`/test pj-rehearse`
ci/rehearse/kube-reporting/metering-operator/release-4.10/ci-index	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/kube-reporting/hive/release-4.10/images	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-shared-resource-operator/release-4.10/e2e-aws	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-cluster-api-provider-kubevirt-release-4.10-sanity-ovn	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-kubevirt-hyperconverged-cluster-operator-release-4.10-hco-e2e-deploy-nightly-main-aws	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-shared-resource/release-4.10/e2e-aws-operator	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-shared-resource/release-4.10/e2e-aws-operator-disruptive	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/openshift/configmap-reload/release-4.10/e2e-aws	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/openshift/csi-driver-shared-resource/release-4.10/e2e-aws-operator-slow	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/openshift/service-ca-operator/release-4.10/e2e-aws-operator	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-cluster-api-provider-kubevirt-release-4.10-sanity-sdn	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/openshift/service-ca-operator/release-4.10/e2e-aws-upgrade	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/openshift/service-ca-operator/release-4.10/e2e-aws	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-ovn-kubernetes-release-4.10-e2e-metal-ipi-ovn-dualstack-periodic	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/kube-reporting/metering-operator/release-4.10/metering-e2e-aws	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-cluster-api-provider-ovirt-release-4.10-e2e-ovirt-prfinder	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-cluster-api-provider-ovirt-release-4.10-e2e-ovirt-cleanup	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-insights-operator-release-4.10-insights-operator-test-time-periodic	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-insights-operator-release-4.10-insights-operator-e2e-tests-periodic	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-kubernetes-release-4.10-k8s-conformance	`fc2dd83`	link	`/test pj-rehearse`
ci/rehearse/periodic-ci-openshift-windows-machine-config-operator-release-4.10-vsphere-e2e-periodic	`fc2dd83`	link	`/test pj-rehearse`
ci/prow/pj-rehearse	`fc2dd83`	link	`/test pj-rehearse`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

petr-muller · 2021-08-04T20:05:07Z

Failing rehearsals are actually hitting a strange pj-rehearse corner case: all these jobs depend on some content coming from tag_specification (=ocp/4.10 imagestream), which is not yet entirely synced to build farms until we actually have an existing config that use it. So this PR shoudl be fine to merge.

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 30, 2021

openshift-ci Bot requested review from abhat and abhinavdahiya June 30, 2021 18:00

petr-muller mentioned this pull request Jun 30, 2021

[4.9]: Carry over manual modifications from master/main jobs #19862

Merged

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 30, 2021

openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 1, 2021

wking mentioned this pull request Jul 1, 2021

ci-operator/jobs/openshift/origin: Restore mutable properties for 4.8 and 4.9 #19716

Merged

petr-muller force-pushed the 410-configs-from-master branch from 7bf002c to 7f87154 Compare August 3, 2021 13:22

openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 3, 2021

openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 3, 2021

petr-muller added 6 commits August 4, 2021 15:18

[4.10] Generate ci-operator configuration

92164b8

[4.10] Generate ci-operator configuration for openshift-priv

d9a7adf

[4.10] Generate vanilla jobs for 4.10

3c3171c

[4.10] Carry job customization over from master/main jobs

ff05dac

[4.10]: Update template allowlist

327a5c9

[4.10]: Bump future release in autoconfigbrancher

fc2dd83

petr-muller force-pushed the 410-configs-from-master branch from 8e6aa69 to fc2dd83 Compare August 4, 2021 13:26

openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 4, 2021

openshift-merge-robot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Aug 4, 2021

openshift-ci Bot merged commit b41c0c4 into openshift:master Aug 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[4.10] Generate ci-operator configuration and jobs#19861

[4.10] Generate ci-operator configuration and jobs#19861
openshift-ci[bot] merged 6 commits intoopenshift:masterfrom
petr-muller:410-configs-from-master

petr-muller commented Jun 30, 2021

Uh oh!

openshift-ci Bot commented Jun 30, 2021

Uh oh!

petr-muller commented Jun 30, 2021

Uh oh!

petr-muller commented Jul 1, 2021

Uh oh!

petr-muller commented Jul 1, 2021 •

edited

Loading

Uh oh!

petr-muller commented Aug 3, 2021

Uh oh!

petr-muller commented Aug 3, 2021

Uh oh!

petr-muller commented Aug 3, 2021

Uh oh!

openshift-ci Bot commented Aug 4, 2021

Uh oh!

petr-muller commented Aug 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

petr-muller commented Jun 30, 2021

Uh oh!

openshift-ci Bot commented Jun 30, 2021

Uh oh!

petr-muller commented Jun 30, 2021

Uh oh!

petr-muller commented Jul 1, 2021

Uh oh!

petr-muller commented Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petr-muller commented Aug 3, 2021

Uh oh!

petr-muller commented Aug 3, 2021

Uh oh!

petr-muller commented Aug 3, 2021

Uh oh!

openshift-ci Bot commented Aug 4, 2021

Uh oh!

petr-muller commented Aug 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

petr-muller commented Jul 1, 2021 •

edited

Loading