Add periodic job for OCP with realtime workers by slintes · Pull Request #7154 · openshift/release

slintes · 2020-02-13T12:12:04Z

Recently a new feature was introduced for MachineConfigs to install a realtime kernel on nodes, see [0] and [1]. To detect regressions with the rt kernel early, this PR introduces a new periodic job.

Todos (AFAIK):

Create a new periodic job for OCP 4.4, using a new CLUSTER_VARIANT "rt" The job needs to run on GCP, as the rt kernel does not work on all AWS machine types.
Handle the new variant in cluster-launch-installer-e2e.yaml
When the job works, add it to release-ocp-4.4.json
When that also works, repeat for OCP 4.5 and 4.6

[0] openshift/enhancements#166
[1] openshift/machine-config-operator#1330

FYI @MarSik

slintes · 2020-02-13T14:26:26Z

Nice, it seems to work :) In the ci/rehearse/release-openshift-ocp-installer-e2e-gcp-rt-4.4 job I see:

the realtime kernel setting in the rendered worker MachineConfig
kernel: Linux version 4.18.0-147.5.1.rt24.98.el8_1.x86_64 in workers journal

But I also see 5 failed tests... are those flaky, or can it be a real issue?

slintes · 2020-02-13T14:36:48Z

old run: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/7154/rehearse-7154-release-openshift-ocp-installer-e2e-gcp-rt-4.4/2

/test ci/rehearse/release-openshift-ocp-installer-e2e-gcp-rt-4.4

slintes · 2020-02-13T15:21:31Z

/test pj-rehearse

slintes · 2020-02-13T17:44:21Z

2nd run had different failures, so I assume they are flakes.
I added jobs for 4.5 and 4.6, and added all to the relevant release json configs.
That's all I know I have to do, please let me know if anything is missing, thanks!

slintes · 2020-02-13T17:45:15Z

/test all

slintes · 2020-02-14T08:28:35Z

/test pj-rehearse

slintes · 2020-02-14T10:30:47Z

looking at other jobs, the unable to import latest release image: timed out waiting for the condition failure seems to be normal on 4.5 and 4.6?

cgwalters · 2020-02-14T14:05:40Z

/approve

slintes · 2020-02-20T18:21:46Z

/assign @smarterclayton

slintes · 2020-02-24T09:01:18Z

ping @smarterclayton, can I get a review please :)

jstuever · 2020-02-26T22:06:17Z

nit: this should probably be flattened out:

if [[ -n "${CLUSTER_NETWORK_MANIFEST:-}" ]] || has_variant "rt"; then openshift-install --dir=/tmp/artifacts/installer/ create manifests fi if [[ -n "${CLUSTER_NETWORK_MANIFEST:-}" ]]; then echo ... fi if has_variant "rt"; then cat > ... fi

or at least untangled:

if [[ -n "${CLUSTER_NETWORK_MANIFEST:-}" ]]; then openshift-install... echo ... fi if has_variant "rt"; then openshift-install... cat >... fi

Thanks for your review!

About your 1st suggestion: not sure, doesn't it make more sense to deal with both features which need to modify manifests inside the if branch which creates them?

About the 2nd: that might probably break if both features are used simultaneously, it would call create manifests twice.

Yes, keep the variant separate. CLUSTER_NETWORK_MANIFEST is a highly specialized use case and support is limited.

split them, and then insight has_variant "rt" have an error condition that fails if CLUSTER_NETWORK_MANIFEST is set:

if has_variant "rt"; then if [[ -n "{CLUSTER_NETWORK_MANIFEST:-}" ]]; then echo 'error: CLUSTER_NETWORK_MANIFEST is incompatible with the `rt` variant' exit 1 fi

jstuever · 2020-02-26T22:07:40Z

/lgtm

jwforres · 2020-03-09T16:03:29Z

/test pj-rehearse

jwforres · 2020-03-09T16:16:41Z

I think you got bit by a rebase issue, you should just have release-pull-secret, rename all references to pull-secret in this PR to release-pull-secret and drop the dupes. There was a rename here 91ef459

oh ha, thanks, fixed

slintes · 2020-03-09T18:42:53Z

The e2e-gcp-rt-4.4 lane looks good, and surprisingly also the e2e-gcp-rt-4.5 lane.
Looking into the app-ci-config failure...

For this a new CLUSTER_VARIANT "rt" is introduced

slintes · 2020-03-10T14:15:02Z

/test app-ci-config

slintes · 2020-03-10T14:17:04Z

/test pj-rehearse

slintes · 2020-03-10T15:51:48Z

ci/rehearse/release-openshift-ocp-installer-e2e-gcp-rt-4.4 is green with 5 test failures
ci/rehearse/release-openshift-ocp-installer-e2e-gcp-rt-4.5 is red but with only 6 failures
ci/rehearse/release-openshift-ocp-installer-e2e-gcp-rt-4.6 failed early with unable to import latest release image: timed out waiting for the condition, is this expected?

ci/rehearse/openshift/cloud-credential-operator/master/e2e-azure still running atm, but the job is unchanged and seems to be red usually anyway (looking at https://prow.svc.ci.openshift.org/?job=*master-e2e-azure)

slintes · 2020-03-16T11:53:25Z

/test all

slintes · 2020-03-16T12:19:17Z

/test pj-rehearse

slintes · 2020-03-16T13:48:20Z

/retest

slintes · 2020-03-16T17:51:04Z

gcp-rt-4.6 still fails early, but other gcp-4.6 lanes fail with the same error.
Everything else looks good from my POV.

Signed-off-by: Marc Sluiter <msluiter@redhat.com>

slintes · 2020-03-16T19:17:32Z

/test pj-rehearse

smarterclayton · 2020-03-16T21:10:35Z

/lgtm

the rt-4.5 job appeared to boot into the right worker kernel

openshift-ci-robot · 2020-03-16T21:11:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, jstuever, slintes, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~ci-operator/jobs/openshift/release/OWNERS~~ [smarterclayton]
~~ci-operator/templates/openshift/installer/OWNERS~~ [smarterclayton]
~~core-services/release-controller/OWNERS~~ [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-03-16T21:33:39Z

@slintes: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/rehearse/release-openshift-ocp-installer-e2e-gcp-rt-4.6	`05e0471`	link	`/test pj-rehearse`
ci/rehearse/openshift/cloud-credential-operator/master/e2e-gcp	`05e0471`	link	`/test pj-rehearse`
ci/rehearse/release-openshift-ocp-installer-e2e-gcp-rt-4.5	`05e0471`	link	`/test pj-rehearse`
ci/rehearse/openshift/cloud-credential-operator/master/e2e-azure	`05e0471`	link	`/test pj-rehearse`
ci/prow/pj-rehearse	`05e0471`	link	`/test pj-rehearse`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 13, 2020

openshift-ci-robot requested review from jstuever and patrickdillon February 13, 2020 12:12

slintes force-pushed the add-realtime-job branch from 0b71d9a to f440c87 Compare February 13, 2020 12:21

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 13, 2020

slintes changed the title ~~[WIP] Add periodic job for OCP with realtime workers~~ Add periodic job for OCP with realtime workers Feb 13, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 13, 2020

slintes requested a review from smarterclayton February 13, 2020 17:48

openshift-ci-robot assigned smarterclayton Feb 20, 2020

jstuever reviewed Feb 26, 2020

View reviewed changes

openshift-ci-robot assigned jstuever Feb 26, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 26, 2020

slintes force-pushed the add-realtime-job branch from cd21e2b to 4199163 Compare March 9, 2020 12:29

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Mar 9, 2020

jwforres reviewed Mar 9, 2020

View reviewed changes

slintes force-pushed the add-realtime-job branch from 24f3532 to fc3cfd1 Compare March 9, 2020 16:53

slintes mentioned this pull request Mar 9, 2020

Revert "Temporarily disable RT kernel" openshift-kni/performance-addon-operators#136

Merged

slintes added 3 commits March 9, 2020 19:48

Added periodic job for OCP 4.4 with realtime workers

2c7cc0c

For this a new CLUSTER_VARIANT "rt" is introduced

Added periodic jobs for OCP 4.5 and 4.6, added all to release configs

46000a3

Apply config changes after rebase

d83c779

slintes force-pushed the add-realtime-job branch from fc3cfd1 to d83c779 Compare March 9, 2020 18:56

jstuever removed their assignment Mar 13, 2020

separated CLUSTER_NETWORK_MANIFEST and rt variant flows

05e0471

Signed-off-by: Marc Sluiter <msluiter@redhat.com>

slintes force-pushed the add-realtime-job branch from 4cc8ac0 to 05e0471 Compare March 16, 2020 18:45

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 16, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 16, 2020

openshift-merge-robot merged commit 0258807 into openshift:master Mar 16, 2020

slintes deleted the add-realtime-job branch June 11, 2024 15:33

Conversation

slintes commented Feb 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slintes commented Feb 13, 2020

Uh oh!

slintes commented Feb 13, 2020

Uh oh!

slintes commented Feb 13, 2020

Uh oh!

slintes commented Feb 13, 2020

Uh oh!

slintes commented Feb 13, 2020

Uh oh!

slintes commented Feb 14, 2020

Uh oh!

slintes commented Feb 14, 2020

Uh oh!

cgwalters commented Feb 14, 2020

Uh oh!

slintes commented Feb 20, 2020

Uh oh!

slintes commented Feb 24, 2020

Uh oh!

jstuever Feb 26, 2020

Choose a reason for hiding this comment

Uh oh!

slintes Feb 27, 2020

Choose a reason for hiding this comment

Uh oh!

smarterclayton Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

smarterclayton Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

slintes Mar 16, 2020

Choose a reason for hiding this comment

Uh oh!

jstuever commented Feb 26, 2020

Uh oh!

jwforres commented Mar 9, 2020

Uh oh!

jwforres Mar 9, 2020

Choose a reason for hiding this comment

Uh oh!

slintes Mar 9, 2020

Choose a reason for hiding this comment

Uh oh!

slintes commented Mar 9, 2020

Uh oh!

slintes commented Mar 10, 2020

Uh oh!

slintes commented Mar 10, 2020

Uh oh!

slintes commented Mar 10, 2020

Uh oh!

slintes commented Mar 16, 2020

Uh oh!

slintes commented Mar 16, 2020

Uh oh!

slintes commented Mar 16, 2020

Uh oh!

slintes commented Mar 16, 2020

Uh oh!

slintes commented Mar 16, 2020

Uh oh!

smarterclayton commented Mar 16, 2020

Uh oh!

openshift-ci-robot commented Mar 16, 2020

Uh oh!

openshift-ci-robot commented Mar 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

slintes commented Feb 13, 2020 •

edited

Loading