From 3bd5e50f19c6076993971b5dbd51469f00fe0e59 Mon Sep 17 00:00:00 2001 From: Pierre Prinetti Date: Mon, 23 Mar 2020 11:24:49 +0100 Subject: [PATCH 1/6] Bug 1815133: osp UPI machineset OS image name The UPI documentation recommends keeping the worker Machineset in case the user wants to create compute machines via the machine API. However, the Machineset won't work unless the `image` property is updated to the user-defined value. This change adds a recommendation to update the OS image in case the user wants to use the installer-provisioned Machineset. --- docs/user/openstack/install_upi.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/user/openstack/install_upi.md b/docs/user/openstack/install_upi.md index 7b7b977849a..32f8e427933 100644 --- a/docs/user/openstack/install_upi.md +++ b/docs/user/openstack/install_upi.md @@ -350,7 +350,9 @@ Remove the control-plane Machines and compute MachineSets, because we'll be prov ```sh $ rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml openshift/99_openshift-cluster-api_worker-machineset-*.yaml ``` -You are free to leave the compute MachineSets in if you want to create compute machines via the machine API, but if you do you may need to update the various references (`subnet`, etc.) to match your environment. +Leave the compute MachineSets in if you want to create compute machines via the machine API. However, some references must be updated in the machineset spec (`openshift/99_openshift-cluster-api_worker-machineset-0.yaml`) to match your environment: + +* The OS image: `spec.template.spec.providerSpec.value.image` [mao]: https://github.com/openshift/machine-api-operator From c3bd6e127fc4604ab816a8cc9713a92c31401252 Mon Sep 17 00:00:00 2001 From: Pierre Prinetti Date: Mon, 30 Mar 2020 13:21:26 +0200 Subject: [PATCH 2/6] Bug 1814651: os UPI known issues: stale resources Following objects are still present in the cluster, after deletion is completed: - Cinder volumes from PVs - Swift container for image registry (bootstrap container is correctly deleted) --- docs/user/openstack/install_upi.md | 3 +++ docs/user/openstack/known-issues.md | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/docs/user/openstack/install_upi.md b/docs/user/openstack/install_upi.md index 32f8e427933..ec052e80dd3 100644 --- a/docs/user/openstack/install_upi.md +++ b/docs/user/openstack/install_upi.md @@ -12,6 +12,9 @@ This provides a greater flexibility at the cost of a more explicit and interacti Below is a step-by-step guide to a UPI installation that mimics an automated IPI installation; prerequisites and steps described below should be adapted to the constraints of the target infrastructure. +Please be aware of the [Known Issues](known-issues.md#known-issues-specific-to-user-provisioned-installations) +of this method of installation. + ## Table of Contents * [Prerequisites](#prerequisites) diff --git a/docs/user/openstack/known-issues.md b/docs/user/openstack/known-issues.md index 38da9680d3e..4dc0efe534d 100644 --- a/docs/user/openstack/known-issues.md +++ b/docs/user/openstack/known-issues.md @@ -30,3 +30,13 @@ Some OpenStack clouds do not set default DNS servers for the newly created subne If you are having this problem in the IPI installer, you will need to set the [`externalDNS` property in `install-config.yaml`](./customization.md#cluster-scoped-properties). Alternatively, for UPI, you will need to [set the subnet DNS resolvers](./install_upi.md#subnet-dns-optional). + +# Known Issues specific to User-Provisioned Installations + +## Stale resources + +The teardown playbooks provided for UPI installation will not delete: + - Cinder volumes from PVs + - Swift container for image registry (bootstrap container is correctly deleted) + +These objects have to be manually removed after running the teardown playbooks. From c60e6b75f4ef5a47be7ec663c91f423ecae29d44 Mon Sep 17 00:00:00 2001 From: Pierre Prinetti Date: Mon, 30 Mar 2020 13:00:17 +0200 Subject: [PATCH 3/6] [release-4.4] Bug 1818608: os UPI fix step order The Boostrap Ignition file needs to be edited before it's uploaded. Before this change, the documentation described the "upload" step before the "edit" step. --- docs/user/openstack/install_upi.md | 140 +++++++++++++++-------------- 1 file changed, 71 insertions(+), 69 deletions(-) diff --git a/docs/user/openstack/install_upi.md b/docs/user/openstack/install_upi.md index ec052e80dd3..4ee60729b10 100644 --- a/docs/user/openstack/install_upi.md +++ b/docs/user/openstack/install_upi.md @@ -413,7 +413,77 @@ Make sure your shell session has the `$INFRA_ID` environment variable set when y ### Bootstrap Ignition -The generated boostrap ignition file (`bootstrap.ign`) tends to be quite large (around 300KB -- it contains all the manifests, master and worker ignitions etc.). This is generally too big to be passed to the server directly (the OpenStack Nova user data limit is 64KB). +#### Edit the Bootstrap Ignition + +We need to set the bootstrap hostname explicitly, and in the case of OpenStack using self-signed certificate, the CA cert file. The IPI installer does this automatically, but for now UPI does not. + +We will update the ignition file (`bootstrap.ign`) to create the following files: + +**`/etc/hostname`**: + +```plaintext +openshift-qlvwv-bootstrap +``` + +(using the `infraID`) + +**`/opt/openshift/tls/cloud-ca-cert.pem`** (if applicable). + +**NOTE**: We recommend you back up the Ignition files before making any changes! + +You can edit the Ignition file manually or run this Python script: + +```python +import base64 +import json +import os + +with open('bootstrap.ign', 'r') as f: + ignition = json.load(f) + +files = ignition['storage'].get('files', []) + +infra_id = os.environ.get('INFRA_ID', 'openshift').encode() +hostname_b64 = base64.standard_b64encode(infra_id + b'-bootstrap\n').decode().strip() +files.append( +{ + 'path': '/etc/hostname', + 'mode': 420, + 'contents': { + 'source': 'data:text/plain;charset=utf-8;base64,' + hostname_b64, + 'verification': {} + }, + 'filesystem': 'root', +}) + +ca_cert_path = os.environ.get('OS_CACERT', '') +if ca_cert_path: + with open(ca_cert_path, 'r') as f: + ca_cert = f.read().encode() + ca_cert_b64 = base64.standard_b64encode(ca_cert).decode().strip() + + files.append( + { + 'path': '/opt/openshift/tls/cloud-ca-cert.pem', + 'mode': 420, + 'contents': { + 'source': 'data:text/plain;charset=utf-8;base64,' + ca_cert_b64, + 'verification': {} + }, + 'filesystem': 'root', + }) + +ignition['storage']['files'] = files; + +with open('bootstrap.ign', 'w') as f: + json.dump(ignition, f) +``` + +Feel free to make any other changes. + +#### Upload the Boostrap Ignition + +The generated boostrap ignition file tends to be quite large (around 300KB -- it contains all the manifests, master and worker ignitions etc.). This is generally too big to be passed to the server directly (the OpenStack Nova user data limit is 64KB). To boot it up, we will create a smaller Ignition file that will be passed to Nova as user data and that will download the main ignition file upon execution. @@ -587,74 +657,6 @@ The result shim config should look like: } ``` -### Update Bootstrap Ignition - -We need to set the bootstrap hostname explicitly, and in the case of OpenStack using self-signed certificate, the CA cert file. The IPI installer does this automatically, but for now UPI does not. - -We will update the ignition to create the following files: - -**`/etc/hostname`**: - -```plaintext -openshift-qlvwv-bootstrap -``` - -(using the `infraID`) - -**`/opt/openshift/tls/cloud-ca-cert.pem`** (if applicable). - -**NOTE**: We recommend you back up the Ignition files before making any changes! - -You can edit the Ignition file manually or run this Python script: - -```python -import base64 -import json -import os - -with open('bootstrap.ign', 'r') as f: - ignition = json.load(f) - -files = ignition['storage'].get('files', []) - -infra_id = os.environ.get('INFRA_ID', 'openshift').encode() -hostname_b64 = base64.standard_b64encode(infra_id + b'-bootstrap\n').decode().strip() -files.append( -{ - 'path': '/etc/hostname', - 'mode': 420, - 'contents': { - 'source': 'data:text/plain;charset=utf-8;base64,' + hostname_b64, - 'verification': {} - }, - 'filesystem': 'root', -}) - -ca_cert_path = os.environ.get('OS_CACERT', '') -if ca_cert_path: - with open(ca_cert_path, 'r') as f: - ca_cert = f.read().encode() - ca_cert_b64 = base64.standard_b64encode(ca_cert).decode().strip() - - files.append( - { - 'path': '/opt/openshift/tls/cloud-ca-cert.pem', - 'mode': 420, - 'contents': { - 'source': 'data:text/plain;charset=utf-8;base64,' + ca_cert_b64, - 'verification': {} - }, - 'filesystem': 'root', - }) - -ignition['storage']['files'] = files; - -with open('bootstrap.ign', 'w') as f: - json.dump(ignition, f) -``` - -Feel free to make any other changes. - ### Master Ignition Similar to bootstrap, we need to make sure the hostname is set to the expected value (it must match the name of the Nova server exactly). From 2daefc834b02cf6660146d180b9ecc26482bdb4b Mon Sep 17 00:00:00 2001 From: Mike Fedosin Date: Thu, 26 Mar 2020 19:33:45 +0100 Subject: [PATCH 4/6] Bug 1816155: Add retries to DeleteGlanceImage Now if the function fails, we stop the installation immediately, but it's better to retry several times before finally stopping the installation. --- pkg/destroy/openstack/glance.go | 36 ++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/pkg/destroy/openstack/glance.go b/pkg/destroy/openstack/glance.go index a94525934e4..fe510d9364f 100644 --- a/pkg/destroy/openstack/glance.go +++ b/pkg/destroy/openstack/glance.go @@ -1,19 +1,42 @@ package openstack import ( + "time" + "github.com/gophercloud/gophercloud/openstack/imageservice/v2/images" "github.com/gophercloud/utils/openstack/clientconfig" + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + + "k8s.io/apimachinery/pkg/util/wait" ) // DeleteGlanceImage deletes the image with the specified name func DeleteGlanceImage(name string, cloud string) error { + backoffSettings := wait.Backoff{ + Duration: time.Second * 20, + Steps: 30, + } + + err := wait.ExponentialBackoff(backoffSettings, func() (bool, error) { + return deleteGlanceImage(name, cloud) + }) + if err != nil { + return errors.Errorf("Unrecoverable error/timed out: %v", err) + } + + return nil +} + +func deleteGlanceImage(name string, cloud string) (bool, error) { opts := clientconfig.ClientOpts{ Cloud: cloud, } conn, err := clientconfig.NewServiceClient("image", &opts) if err != nil { - return err + logrus.Warningf("There was an error during the image removal: %v", err) + return false, nil } listOpts := images.ListOpts{ @@ -22,19 +45,22 @@ func DeleteGlanceImage(name string, cloud string) error { allPages, err := images.List(conn, listOpts).AllPages() if err != nil { - return err + logrus.Warningf("There was an error during the image removal: %v", err) + return false, nil } allImages, err := images.ExtractImages(allPages) if err != nil { - return err + logrus.Warningf("There was an error during the image removal: %v", err) + return false, nil } for _, image := range allImages { err := images.Delete(conn, image.ID).ExtractErr() if err != nil { - return err + logrus.Warningf("There was an error during the image removal: %v", err) + return false, nil } } - return nil + return true, nil } From 8dcc11a03e2ae794a004aa0a176d2e2951602531 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Sun, 29 Mar 2020 17:40:56 +0300 Subject: [PATCH 5/6] ovirt - fix the credentials folder permissions Signed-off-by: Roy Golan --- pkg/asset/installconfig/ovirt/config.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/asset/installconfig/ovirt/config.go b/pkg/asset/installconfig/ovirt/config.go index afd02bbe734..656fbbf851d 100644 --- a/pkg/asset/installconfig/ovirt/config.go +++ b/pkg/asset/installconfig/ovirt/config.go @@ -68,9 +68,9 @@ func (c *Config) Save() error { } path := discoverPath() - err = os.MkdirAll(filepath.Dir(path), os.FileMode(700)) + err = os.MkdirAll(filepath.Dir(path), 0700) if err != nil { return err } - return ioutil.WriteFile(path, out, os.FileMode(0600)) + return ioutil.WriteFile(path, out, 0600) } From a29835acda923d423ca26caa9ad68ec38ef4b2ab Mon Sep 17 00:00:00 2001 From: Vadim Rutkovsky Date: Tue, 14 Apr 2020 14:39:03 +0200 Subject: [PATCH 6/6] Bump MCO --- go.mod | 2 +- go.sum | 7 ++++--- vendor/modules.txt | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/go.mod b/go.mod index cc874de5881..f5b191866ac 100644 --- a/go.mod +++ b/go.mod @@ -135,7 +135,7 @@ replace ( github.com/metal3-io/baremetal-operator => github.com/openshift/baremetal-operator v0.0.0-20200206190020-71b826cc0f0a // Use OpenShift fork github.com/metal3-io/cluster-api-provider-baremetal => github.com/openshift/cluster-api-provider-baremetal v0.0.0-20190821174549-a2a477909c1d // Pin OpenShift fork github.com/openshift/api => github.com/openshift/api v0.0.0-20200210091934-a0e53e94816b // Pin API - github.com/openshift/machine-config-operator => github.com/LorbusChris/machine-config-operator v0.0.1-0.20200323222512-db1a46cb4bbb // Pin FCOS MCO + github.com/openshift/machine-config-operator => github.com/openshift/machine-config-operator v0.0.1-0.20200414120921-1de18b18146b // Pin FCOS MCO github.com/terraform-providers/terraform-provider-azurerm => github.com/openshift/terraform-provider-azurerm v1.41.1-openshift-3 // Pin to openshift fork with IPv6 fixes github.com/terraform-providers/terraform-provider-ignition/v2 => github.com/LorbusChris/terraform-provider-ignition/v2 v2.0.0-20200118034038-6e413297dc57 google.golang.org/api => google.golang.org/api v0.13.0 // Pin to version required by tf-provider-google diff --git a/go.sum b/go.sum index 0fc5ad8b9dc..e231c86fd04 100644 --- a/go.sum +++ b/go.sum @@ -105,8 +105,6 @@ github.com/InVisionApp/go-health v2.1.0+incompatible/go.mod h1:/+Gv1o8JUsrjC6pi6 github.com/JeffAshton/win_pdh v0.0.0-20161109143554-76bb4ee9f0ab/go.mod h1:3VYc5hodBMJ5+l/7J4xAyMeuM2PNuepvHlGs8yilUCA= github.com/Jeffail/gabs v1.1.1/go.mod h1:6xMvQMK4k33lb7GUUpaAPh6nKMmemQeg5d4gn7/bOXc= github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible/go.mod h1:r7JcOSlj0wfOMncg0iLm8Leh48TZaKVeNIfJntJ2wa0= -github.com/LorbusChris/machine-config-operator v0.0.1-0.20200323222512-db1a46cb4bbb h1:zZhZOgHnNq3Zj65UA63NIGcUPGAIGrzSae/QFAm0ia8= -github.com/LorbusChris/machine-config-operator v0.0.1-0.20200323222512-db1a46cb4bbb/go.mod h1:EnsJFPg9wiFS06ORFuhuQfgWQCzFbctj15jW9i2o5PU= github.com/LorbusChris/terraform-provider-ignition/v2 v2.0.0-20200118034038-6e413297dc57 h1:BupptCmXjF41p4eUw+yBwwyHgovRAkAt/hH7PDk1u8U= github.com/LorbusChris/terraform-provider-ignition/v2 v2.0.0-20200118034038-6e413297dc57/go.mod h1:kjYnSlonNzYk08tQnHAh0e/9e/GV7tM8ClGwPCQJqqI= github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd/go.mod h1:64YHyfSL2R96J44Nlwm39UHepQbyR5q10x7iYa1ks2E= @@ -371,7 +369,7 @@ github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc github.com/coreos/etcd v3.3.15+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/etcd v3.3.18+incompatible h1:Zz1aXgDrFFi1nadh58tA9ktt06cmPTwNNP3dXwIq1lE= github.com/coreos/etcd v3.3.18+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= -github.com/coreos/fcct v0.4.0/go.mod h1:rNqhmWqBE0ArcrunuaCz/+P2FHcDHlaOcgQUDOPr5oQ= +github.com/coreos/fcct v0.5.0/go.mod h1:cbE+j77YSQwFB2fozWVB3qsI2Pi3YiVEbDz/b6Yywdo= github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk= github.com/coreos/go-iptables v0.4.2/go.mod h1:/mVI274lEDI2ns62jHCDnCyBF9Iwsmekav8Dbxlm1MU= github.com/coreos/go-oidc v2.0.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc= @@ -385,6 +383,7 @@ github.com/coreos/ignition v0.33.0/go.mod h1:WJQapxzEn9DE0ryxsGvm8QnBajm/XsS/Pkr github.com/coreos/ignition v0.34.0 h1:r3HQKhQmRDhofcRSFp2fDVB6vhqb9gubEnyjgM+l3GQ= github.com/coreos/ignition v0.34.0/go.mod h1:WJQapxzEn9DE0ryxsGvm8QnBajm/XsS/PkrDqSpz+bA= github.com/coreos/ignition/v2 v2.1.1/go.mod h1:RqmqU64zxarUJa3l4cHtbhcSwfQLpUhv0WVziZwoXvE= +github.com/coreos/ignition/v2 v2.2.1/go.mod h1:RqmqU64zxarUJa3l4cHtbhcSwfQLpUhv0WVziZwoXvE= github.com/coreos/ignition/v2 v2.2.2-0.20200325194711-7d770711ebc1 h1:kEZ+EZv6PVXHZ1JnzvFePPTP7zNpUhB/h/hos1GUw20= github.com/coreos/ignition/v2 v2.2.2-0.20200325194711-7d770711ebc1/go.mod h1:RqmqU64zxarUJa3l4cHtbhcSwfQLpUhv0WVziZwoXvE= github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= @@ -1811,6 +1810,8 @@ github.com/openshift/machine-api-operator v0.0.0-20190312153711-9650e16c9880/go. github.com/openshift/machine-api-operator v0.2.1-0.20191128180243-986b771e661d/go.mod h1:9qQPF00anuIsc6RiHYfHE0+cZZImbvFNLln0NRBVVMg= github.com/openshift/machine-api-operator v0.2.1-0.20200310180732-c63fa2b143f0 h1:Na0422T5qq9e4AtBqH4hyqujESg29Akrf2asy/kc02U= github.com/openshift/machine-api-operator v0.2.1-0.20200310180732-c63fa2b143f0/go.mod h1:b3huCV+DbroXP1sHtsU5xBwx97zqc6GKB5owyl2zsNM= +github.com/openshift/machine-config-operator v0.0.1-0.20200414120921-1de18b18146b h1:0P/nbeROvQg02urwzqeQ2yKI6L0dluyYnT0n2XnYJmw= +github.com/openshift/machine-config-operator v0.0.1-0.20200414120921-1de18b18146b/go.mod h1:yZV361656gvAH+Tg6izYvSiH9reMGIP3lcB04ip8IRc= github.com/openshift/runtime-utils v0.0.0-20191011150825-9169de69ebf6/go.mod h1:5gDRVvQwesU7cfwlpuMivdv3Dz/oslvv2qTBHCy4wqQ= github.com/openshift/terraform-provider-azurerm v1.41.1-openshift-3 h1:aRnSZYFNqYXv8mc3/q6nP1WJP4VR8eugkdmJF6xnCs0= github.com/openshift/terraform-provider-azurerm v1.41.1-openshift-3/go.mod h1:O9UYVGp8E7aMa2dANw9oTQmaZZbbr8DMRpC56dyy00E= diff --git a/vendor/modules.txt b/vendor/modules.txt index b32fba8c8c3..5d3d0f737d1 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -1037,7 +1037,7 @@ github.com/openshift/machine-api-operator/pkg/apis/machine github.com/openshift/machine-api-operator/pkg/apis/machine/v1beta1 github.com/openshift/machine-api-operator/pkg/apis/vsphereprovider github.com/openshift/machine-api-operator/pkg/apis/vsphereprovider/v1alpha1 -# github.com/openshift/machine-config-operator v4.2.0-alpha.0.0.20190917115525-033375cbe820+incompatible => github.com/LorbusChris/machine-config-operator v0.0.1-0.20200323222512-db1a46cb4bbb +# github.com/openshift/machine-config-operator v4.2.0-alpha.0.0.20190917115525-033375cbe820+incompatible => github.com/openshift/machine-config-operator v0.0.1-0.20200414120921-1de18b18146b github.com/openshift/machine-config-operator/pkg/apis/machineconfiguration.openshift.io/v1 # github.com/ovirt/go-ovirt v4.3.4+incompatible github.com/ovirt/go-ovirt