[FCOS] Update vSphere UPI by vrutkovs · Pull Request #2671 · openshift/installer

vrutkovs · 2019-11-14T10:01:41Z

Update OVA path in rhcos.json to official Fedora CoreOS.

This uses a forked version of ignition terraform provider to support spec3, updates UPI dockerfile to use it and uses NM keyfiles to replace no longer supported ifcfgs.

This PR also avoids an extra reboot to set static IP - only NM is being restarted instead

TODO:

Requires FCOS installer: add vsphere test release#5914 to have a test added
Bootstrap fails, cannot be ssh'ed to
Masters can't find a route to bootstrap
Add open-vm-tools to machine-os-content so that PVC tests would pass
OKD 4.4: use repos from fedora image and install open-vm-tools for vSphere release#6590
~~Use FQDN in hostname (OVN verifies that) and create necessary Route53 records~~
OVN doesn't use search domain, FQDN name is too long for node name. Use IP address instead

Fixes okd-project/okd#14

hardys · 2019-11-14T12:02:54Z

Will all of the platforms builds be available at this URI? We use this location for IPI baremetal to download/cache the openstack qcow2 image so it'd be good to confirm it that's likely to work :)

https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20191014.0/x86_64/rhcos-43.80.20191002.1-openstack.x86_64.qcow2 403s so I guess the answer is no - will we rename all the filenames to fcos to align with this?

Yes, eventually. So far only Azure and GCP artifacts are not available.

I'm splitting updates into several PRs so that these could be tested and tracked separately

These are xz compressed - https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/30.20191014.0/x86_64/fedora-coreos-30.20191014.0-openstack.x86_64.qcow2.xz, so for openstack/libvirt/baremetal a different PR is required, which implements xz uncompress

Ack #2657 merged which adds gz decompression and a way to plug in xz compression with a new decompress function.

We'll also need to update https://github.com/openshift/ironic-rhcos-downloader to support xz compression.

Ah nice I see you have the installer xz support in-progress via #2678 - I'll push a pr adding xz support to ironic-rhcos-downloader (which is also in the process of being renamed btw)

Yeah, lets focus on vSphere in this PR. Other platforms support would be fixed in followup PRs.

Baremetal needs quite a few fixes:

switch CI to replace 'ironic-*' images with metal3 images (so that we won't release OSP code in OKD)

Use generic name in ironic-rhcos-downloader name (I don't think there is consensus on this yet)

fix links in the installer and setup baremetal OKD CI

vrutkovs · 2019-11-14T16:30:21Z

/cc @smarterclayton @LorbusChris

It seems vsphere support can be added right away

vrutkovs · 2019-11-15T08:17:38Z

/retest

vrutkovs · 2019-11-15T17:32:37Z

/retest

vrutkovs · 2019-11-15T20:34:17Z

/retest

vrutkovs · 2019-11-18T13:39:19Z

/retest

vrutkovs · 2019-11-19T09:16:39Z

/retest

vrutkovs · 2019-11-19T11:44:07Z

/retest

LorbusChris · 2020-01-18T05:52:11Z

/retest

LorbusChris · 2020-01-18T06:04:42Z

/lgtm

LorbusChris · 2020-01-19T00:38:09Z

/retest

vrutkovs · 2020-01-19T09:07:29Z

ovnkube-master gets stuck due to two interfaces:

++ getent ahostsv4 control-plane-2
++ awk '{print $1}'
+ host='139.178.87.133
172.17.0.1'

UPD CI picked up an older cluster-network-operator for some reason

vrutkovs · 2020-01-19T16:10:01Z

time="2020-01-19T16:04:40Z" level=fatal msg="failed to obtain local IP from hostname "compute-0": Failed to lookup IP address for node compute-0: lookup compute-0 on 1.1.1.1:53: no such host"

Seems OVN won't accept fake node names. It uses hostname (as identified by k8s), but it cannot resolve it using upstream DNS

OVN attempts to resolve node name to find out its IP, so it cannot be phoney. OVN, unfortunately, doesn't use search domains, so in order to join the node CI terraform scripts use node IP as a name

LorbusChris · 2020-01-19T23:30:08Z

/lgtm

vrutkovs · 2020-01-19T23:39:04Z

Prometheus when installed on the cluster should have important platform topology metrics test is failing.

/hold cancel

Lets see if this is a flake (afaik even OCP's AWS UPI install is affected)

openshift-ci-robot · 2020-01-19T23:53:05Z

@vrutkovs: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-vsphere	`92030e8`	link	`/test e2e-vsphere`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vrutkovs · 2020-01-19T23:55:47Z

Right, that affects AWS UPI too - https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-upi-4.4/497

/override ci/prow/e2e-vsphere

openshift-ci-robot · 2020-01-19T23:56:09Z

@vrutkovs: Overrode contexts on behalf of vrutkovs: ci/prow/e2e-vsphere

Details

In response to this:

Right, that affects AWS UPI too - https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-upi-4.4/497

/override ci/prow/e2e-vsphere

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 14, 2019

openshift-ci-robot requested review from jcpowermac and mtnbikenc November 14, 2019 10:02

vrutkovs force-pushed the fcos-vmware-path branch from c64110c to e3a2692 Compare November 14, 2019 10:41

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 14, 2019

hardys reviewed Nov 14, 2019

View reviewed changes

openshift-ci-robot requested review from LorbusChris and smarterclayton November 14, 2019 16:30

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 15, 2019

vrutkovs force-pushed the fcos-vmware-path branch from 7d86100 to 3728084 Compare November 15, 2019 13:32

vrutkovs force-pushed the fcos-vmware-path branch 2 times, most recently from 79f42b5 to 1463540 Compare November 18, 2019 13:26

vrutkovs force-pushed the fcos-vmware-path branch 2 times, most recently from 1e88a02 to cac2160 Compare November 19, 2019 10:19

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 19, 2019

vrutkovs force-pushed the fcos-vmware-path branch 2 times, most recently from 9e913e8 to e9efa89 Compare November 19, 2019 15:02

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 19, 2019

vsphere: update to ignition spec3

113d0c8

vrutkovs force-pushed the fcos-vmware-path branch from 9bc09a1 to 9b42df2 Compare January 17, 2020 13:06

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jan 17, 2020

vrutkovs force-pushed the fcos-vmware-path branch from 9b42df2 to e0f09cd Compare January 17, 2020 13:43

Vadim Rutkovsky added 4 commits January 17, 2020 14:54

Use forked terraform-ignition

41eab68

Dockerfile.upi.ci: use custom ignition provider

c9d2516

vSphere: use NM keyfiles instead of deprecated ifcfgs

1d0dca8

vSphere: restart NetworkManager instead of reboot to setup static IP

a8911b2

vrutkovs force-pushed the fcos-vmware-path branch from e0f09cd to a8911b2 Compare January 17, 2020 13:55

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 18, 2020

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jan 19, 2020

vrutkovs force-pushed the fcos-vmware-path branch from 957c960 to a8911b2 Compare January 19, 2020 15:48

vrutkovs force-pushed the fcos-vmware-path branch 2 times, most recently from e89b7a7 to fa30a14 Compare January 19, 2020 20:45

upi/vsphere/machine/ignition: set hostname to IP address

92030e8

OVN attempts to resolve node name to find out its IP, so it cannot be phoney. OVN, unfortunately, doesn't use search domains, so in order to join the node CI terraform scripts use node IP as a name

vrutkovs force-pushed the fcos-vmware-path branch from fa30a14 to 92030e8 Compare January 19, 2020 22:27

vrutkovs mentioned this pull request Jan 19, 2020

[FCOS] use stable Fedora CoreOS boot images #2947

Merged

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 19, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 19, 2020

openshift-merge-robot merged commit be80749 into openshift:fcos Jan 19, 2020

Conversation

vrutkovs commented Nov 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hardys Nov 14, 2019

Choose a reason for hiding this comment

Uh oh!

hardys Nov 14, 2019

Choose a reason for hiding this comment

Uh oh!

vrutkovs Nov 14, 2019

Choose a reason for hiding this comment

Uh oh!

vrutkovs Nov 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hardys Nov 19, 2019

Choose a reason for hiding this comment

Uh oh!

hardys Nov 19, 2019

Choose a reason for hiding this comment

Uh oh!

vrutkovs Nov 19, 2019

Choose a reason for hiding this comment

Uh oh!

vrutkovs commented Nov 14, 2019

Uh oh!

vrutkovs commented Nov 15, 2019

Uh oh!

vrutkovs commented Nov 15, 2019

Uh oh!

vrutkovs commented Nov 15, 2019

Uh oh!

vrutkovs commented Nov 18, 2019

Uh oh!

vrutkovs commented Nov 19, 2019

Uh oh!

vrutkovs commented Nov 19, 2019

Uh oh!

LorbusChris commented Jan 18, 2020

Uh oh!

LorbusChris commented Jan 18, 2020

Uh oh!

LorbusChris commented Jan 19, 2020

Uh oh!

vrutkovs commented Jan 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vrutkovs commented Jan 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LorbusChris commented Jan 19, 2020

Uh oh!

vrutkovs commented Jan 19, 2020

Uh oh!

openshift-ci-robot commented Jan 19, 2020

Uh oh!

vrutkovs commented Jan 19, 2020

Uh oh!

openshift-ci-robot commented Jan 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

vrutkovs commented Nov 14, 2019 •

edited

Loading

vrutkovs Nov 14, 2019 •

edited

Loading

vrutkovs commented Jan 19, 2020 •

edited

Loading

vrutkovs commented Jan 19, 2020 •

edited

Loading