USHIFT-1363: tools to build multiple images for ci by dhellmann · Pull Request #1951 · openshift/microshift

dhellmann · 2023-06-22T22:51:40Z

This PR adds a framework for managing "test scenarios". It is possible to independently define images, VMs, and tests, then mix and match them to produce scenarios like "test upgrade and rollback from MicroShift 4.13 to 4.14 on RHEL 9.2". The PR includes a few scenarios as examples. The scripts have been tested on a RHEL hypervisor using remote access similar to what we will do in CI. The next phase of implementation will include some wrappers and integration scripts for linking in to CI, but these scripts are usable as-is for developers to run locally.

openshift-ci-robot · 2023-06-25T20:34:04Z

@dhellmann: This pull request references USHIFT-1363 which is a valid jira issue.

Details

In response to this:

Which issue(s) this PR addresses:

Closes #

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-06-25T20:36:43Z

@dhellmann: This pull request references USHIFT-1363 which is a valid jira issue.

Details

In response to this:

This PR adds a framework for managing "test scenarios". It is possible to independently define images, VMs, and tests, then mix and match them to produce scenarios like "test upgrade and rollback from MicroShift 4.13 to 4.14 on RHEL 9.2". The PR includes a few scenarios as examples. The scripts have been tested on a RHEL hypervisor using remote access similar to what we will do in CI. The next phase of implementation will include some wrappers and integration scripts for linking in to CI, but these scripts are usable as-is for developers to run locally.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

…stem When we have to get the kubeconfig from a remote system through a port-forwarded connection, we may have to update the connection URL.

…stem When we run the tests on a remote system through a port-forwarded connection, we may have to modify some connection parameters.

pmtk

overall looks good to me, waiting for final version :)

dnf is reporting different missing file errors in CI, so retry the command a few times

Define a test harness for running different "scenarios" defined by combining OS images, MicroShift versions, and test suites.

Add image blueprint data for RHEL 9.2 images without MicroShift and with MicroShift 4.13 and the package built from source. Add package sources for fast-datapath and RHOCP 4.13 for dependencies. Add a kickstart template for booting an image from an ostree commit. Add a scenario for running the standard test suite against a host running RHEL 9.2 and MicroShift built from source. Add a scenario for running the upgrade test against a host running RHEL 9.2 and MicroShift built from source.

pmtk

just some nits we can address in separate PR
/lgtm
/hold

pmtk · 2023-07-03T08:50:01Z

+    local failed=true
+    local deps="libvirt virt-manager virt-install virt-viewer libvirt-client qemu-kvm qemu-img sshpass"
+
+    set +e  # retry because we see errors caching different RPMs in CI


Which failure is this supposed to prevent? I think if command; then doesn't need this

I think I had this here before the linter made me use the if statement. I'll take it out in a follow-up.

pmtk · 2023-07-03T08:58:42Z

+
+# Build all of the needed VMs
+for scenario in scenarios/*.sh; do
+    time bash -x ./bin/scenario.sh create "${scenario}"


time or /usr/bin/time? /usr/bin/time -v prints some nice stats

I was mostly curious how long it was taking to make the VMs, so I just wanted clock time. This section is going to be rewritten to run the steps in parallel, and I can use /usr/bin/time -v when I do that.

pmtk · 2023-07-03T09:24:12Z

+    local ip="${2}"
+
+    echo "Waiting ${VM_BOOT_TIMEOUT} for greenboot on ${vmname} to complete"
+    timeout "${VM_BOOT_TIMEOUT}" bash -c "until ssh redhat@${ip} \"sudo journalctl -n 5 -u greenboot-healthcheck; sudo systemctl status greenboot-healthcheck | grep 'active (exited)'\"; do date; sleep 10; done"


Are we sure -n 5 is enough?

It's in a loop, so it shows the last few lines every 10 seconds. I saw a lot of repeated output. I could make it print more lines if you think that would help? Maybe 10?

I missed the loop :)

openshift-ci · 2023-07-03T09:26:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dhellmann, pmtk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dhellmann,pmtk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dhellmann · 2023-07-03T13:00:59Z

/hold cancel

openshift-ci · 2023-07-03T14:23:16Z

@dhellmann: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-openshift-conformance-reduced-arm	`4860f1a`	link	false	`/test e2e-openshift-conformance-reduced-arm`
ci/prow/microshift-e2e-arm	`4860f1a`	link	false	`/test microshift-e2e-arm`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 22, 2023

openshift-ci Bot requested review from jogeo and pacevedom June 22, 2023 22:52

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 22, 2023

dhellmann marked this pull request as draft June 22, 2023 22:52

dhellmann removed request for jogeo and pacevedom June 22, 2023 23:43

pmtk reviewed Jun 23, 2023

View reviewed changes

Comment thread test/README.md Outdated

Comment thread test/image-blueprints/rhel92-microshift413.toml Outdated

Comment thread test/bin/download_images.sh Outdated

dhellmann force-pushed the build-multiple-images-for-ci branch 10 times, most recently from 72f7727 to f9fc47f Compare June 25, 2023 19:56

dhellmann changed the title ~~WIP: tools to build multiple images for ci~~ USHIFT-1363: tools to build multiple images for ci Jun 25, 2023

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 25, 2023

dhellmann marked this pull request as ready for review June 25, 2023 20:34

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 25, 2023

openshift-ci Bot requested review from copejon and pmtk June 25, 2023 20:34

dhellmann requested review from ggiguash, jogeo and pacevedom June 25, 2023 20:39

dhellmann force-pushed the build-multiple-images-for-ci branch from f9fc47f to d77f20e Compare June 25, 2023 21:10

dhellmann added 3 commits June 29, 2023 17:31

scripts: add caddy as dependency for image builder

eb5fc4b

test: make kubeconfig resource compatible with running on a remote sy…

01566b1

…stem When we have to get the kubeconfig from a remote system through a port-forwarded connection, we may have to update the connection URL.

test: make load-balancer tests compatible with running on a remote sy…

22ec1c1

…stem When we run the tests on a remote system through a port-forwarded connection, we may have to modify some connection parameters.

dhellmann force-pushed the build-multiple-images-for-ci branch from cd89147 to a61f75a Compare June 29, 2023 22:08

pmtk reviewed Jun 30, 2023

View reviewed changes

Comment thread test/README.md Outdated

Comment thread test/bin/ci_phase_iso_boot.sh Outdated

Comment thread test/image-blueprints/rhel92.toml Outdated

dhellmann force-pushed the build-multiple-images-for-ci branch from 931ba69 to 866ce46 Compare June 30, 2023 15:37

add error options to manage-vm.sh for use in ci

782dc0b

dhellmann force-pushed the build-multiple-images-for-ci branch 7 times, most recently from 5ec860a to 7ca02c1 Compare July 1, 2023 21:26

dhellmann added 4 commits July 2, 2023 13:07

manage-vm: retry installing dependencies

1b18a47

dnf is reporting different missing file errors in CI, so retry the command a few times

USHIFT-1363: add multi-vm test harness

36b7361

Define a test harness for running different "scenarios" defined by combining OS images, MicroShift versions, and test suites.

USHIFT-1363: add ci integration scripts for scenario tests

4860f1a

dhellmann force-pushed the build-multiple-images-for-ci branch from f448dc2 to 4860f1a Compare July 2, 2023 17:08

pmtk reviewed Jul 3, 2023

View reviewed changes

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 3, 2023

openshift-ci Bot assigned pmtk Jul 3, 2023

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 3, 2023

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 3, 2023

openshift-merge-robot merged commit 9ef7a6b into openshift:main Jul 3, 2023

This was referenced May 5, 2026

NO-ISSUE: rebase-release-5.0-5.0.0-0.nightly-2026-05-04-160406_amd64-2026-05-04_arm64-2026-05-04 #6623

Closed

NO-ISSUE: rebase-release-5.0-5.0.0-0.nightly-2026-05-04-160406_amd64-2026-05-04_arm64-2026-05-05 #6630

Merged

Conversation

dhellmann commented Jun 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 25, 2023 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 25, 2023 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmtk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmtk left a comment

Choose a reason for hiding this comment

Uh oh!

pmtk Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

dhellmann Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pmtk Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

dhellmann Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pmtk Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

dhellmann Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pmtk Jul 4, 2023

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Jul 3, 2023

Uh oh!

dhellmann commented Jul 3, 2023

Uh oh!

openshift-ci Bot commented Jul 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dhellmann commented Jun 22, 2023 •

edited

Loading

openshift-ci-robot commented Jun 25, 2023 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jun 25, 2023 •

edited by openshift-ci Bot

Loading

pmtk left a comment •

edited

Loading