OVF datasource should check if instant id is still on VMware Platform

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1835205

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = None
date_created = 2019-07-03T11:24:36.299696+00:00
date_fix_committed = None
date_fix_released = None
id = 1835205
importance = medium
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1835205
milestone = None
owner = pengpengs
owner_name = Pengpeng Sun
private = False
status = triaged
submitter = pengpengs
submitter_name = Pengpeng Sun
tags = []
duplicates = []

Launchpad user Pengpeng Sun(pengpengs) wrote on 2019-07-03T11:24:36.299696+00:00

Currently DatasourceOVF does not check the instance id to determine if current instance is a new instance or not, so that every booting, cloud-init will go through entire datasource list.

This leads an issue:
When a VM's network is customized to static IP by cloud-init DatasourceOVF and then the VM is rebooted, the VM's network will be changed to DHCP after rebooting since this time no customization config file will be found. cloud-init uses DatasourceNone which change network to "default" configuration.

The expected behaviors are:

1. cloud-init checks the instance id to know if this instance is "iid-vmware-xxxxxx" which means it's a VMware VM. If yes, it should always use datasourceOVF.
2. When there has customization config file, datasourceOVF parse the configures and enable nics as usual.
3. When there is no customization config file, datasourceOVF wait for it until timeout and then cloud-init does NOT apply other datasources.

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2019-07-18T20:40:03.474066+00:00

Thanks for filling a bug.

Cloud-init does check instance-id by looking in /var/lib/cloud/data/ at files 'instance-id' and 'previous-instance-id' and will check if the existing Datasource's metadata provides the same instance-id.

The behavior you describe sounds like the OVF datasource changed.

Please run cloud-init collect-logs as root and attach the tarball so we can see what's going on.

Thanks!

[ubuntu-server-builder]

Launchpad user Pengpeng Sun(pengpengs) wrote on 2019-07-24T08:04:21.726080+00:00

Hi Ryan,

After cloud-init customization done, I saw /var/lib/cloud/instance links to /var/lib/cloud/instances/iid-vmware-XXXXXXXX. But this instance seems not take effect at next reboot, cloud-init always think it's a new instance. Do a reboot here, the /var/lib/cloud/instance links to /var/lib/cloud/instances/iid-datasource-none

I think check_instance_id is not implemented in OVF datasource.

Thanks,
Pengpeng

[ubuntu-server-builder]

Launchpad user Pengpeng Sun(pengpengs) wrote on 2019-07-24T08:05:01.768839+00:00

Attach logs after cloud-init customization by OVF datasource.
Launchpad attachments: cloud-init-after-customization.tar.gz

[ubuntu-server-builder]

Launchpad user Pengpeng Sun(pengpengs) wrote on 2019-07-24T08:05:29.344092+00:00

Attach logs after reboot.
Launchpad attachments: cloud-init-after-boot.tar.gz

[ubuntu-server-builder]

Launchpad user Ilia(ilia-zenkevich) wrote on 2019-07-24T13:25:31.376996+00:00

Hi all.
Faced with the same issue.

Environment:
vCloud Director 9.1
Ubuntu Server 16.04.1 x64
open-vm-tools 2:10.2.0-3ubuntu0.16.04.1
cloud-init 19.1-1-gbaa47854-0ubuntu116.04.1

cloud-init log for second and later boots always contains

2019-07-23 13:22:39,347 - DataSourceOVF.py[DEBUG]: Did not find VMware Customization Config File
2019-07-23 13:22:39,347 - util.py[DEBUG]: Running command ['vmware-rpctool', 'info-get guestinfo.ovfEnv'] with allowed return codes [0]
(shell=False, capture=True)
2019-07-23 13:22:39,353 - handlers.py[DEBUG]: finish: init-local/search-OVF: FAIL: no local data found from DataSourceOVF
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOVF.py", line 394, in read_ovf_environment
props = get_properties(contents)
File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOVF.py", line 553, in get_properties
raise XmlError("No 'PropertySection's")
cloudinit.sources.DataSourceOVF.XmlError: No 'PropertySection's

If I run "vmware-rpctool info-get guestinfo.ovfEnv" manually, I can't see "PropertySection" sections, but the question is: Is this section something that have to be in the response?

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2019-07-24T14:34:55.308416+00:00

Note that check_instance_id() is only needed if you want to avoid fetching metadata/userdata a second time; it does not prevent cloud-init from knowing it's on the same instance; In all cases, the metadata that indicates the instance-id must remain associated with the instance.

Looking at the logs; it appears that after reboot the Customization file is not found and OVF datasource _get_data() method returns False; which tells cloud-init that it's not using OVF as a datasource.

1. firstboot finds OVF Customication

2019-07-24 07:50:43,682 - util.py[DEBUG]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'init-local' at Wed, 24 Jul 2019 07:50:43 +0000. Up 12.30 seconds.
...
2019-07-24 07:50:43,802 - init.py[DEBUG]: Searching for local data source in: ['DataSourceOVF']
2019-07-24 07:50:43,802 - handlers.py[DEBUG]: start: init-local/search-OVF: searching for local data from DataSourceOVF
2019-07-24 07:50:43,802 - init.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit.sources.DataSourceOVF.DataSourceOVF'>
2019-07-24 07:50:43,802 - init.py[DEBUG]: Update datasource metadata and network config due to events: New instance first boot

2019-07-24 07:50:43,813 - DataSourceOVF.py[DEBUG]: VMware Virtualization Platform found
2019-07-24 07:50:43,813 - DataSourceOVF.py[DEBUG]: Found the customization plugin at /usr/lib/open-vm-tools/plugins/vmsvc/libdeployPkgPlugin.so
2019-07-24 07:50:43,813 - DataSourceOVF.py[DEBUG]: Waiting for VMware Customization Config File
2019-07-24 07:50:48,818 - util.py[DEBUG]: waiting for configuration file took 5.005 seconds
2019-07-24 07:50:48,823 - DataSourceOVF.py[DEBUG]: Found VMware Customization Config File at /var/run/vmware-imc/cust.cfg
2019-07-24 07:50:49,928 - handlers.py[DEBUG]: finish: init-local/search-OVF: SUCCESS: found local data from DataSourceOVF
2019-07-24 07:50:49,929 - stages.py[INFO]: Loaded datasource DataSourceOVF - DataSourceOVF [seed=vmware-tools]

This sets the instance id to:

2019-07-24 07:50:49,945 - util.py[DEBUG]: Creating symbolic link from '/var/lib/cloud/instance' => '/var/lib/cloud/instances/iid-vmware-QokNwWPI'

2. Second boot of same instance does not find metadata from DataSourceOVF

2019-07-24 07:56:49,746 - DataSourceOVF.py[DEBUG]: VMware Virtualization Platform found
2019-07-24 07:56:49,746 - DataSourceOVF.py[DEBUG]: Found the customization plugin at /usr/lib/open-vm-tools/plugins/vmsvc/libdeployPkgPlugin.so
2019-07-24 07:56:49,746 - DataSourceOVF.py[DEBUG]: Waiting for VMware Customization Config File

2019-07-24 07:58:19,856 - util.py[DEBUG]: waiting for configuration file took 90.110 seconds
2019-07-24 07:58:19,858 - DataSourceOVF.py[DEBUG]: Did not find VMware Customization Config File

2019-07-24 07:58:20,152 - main.py[DEBUG]: [local] Exiting without datasource

Why did the customization data for the instance go away?

[ubuntu-server-builder]

Launchpad user Ilia(ilia-zenkevich) wrote on 2019-07-24T16:47:24.325317+00:00

Because cloud-init deletes it in DataSourceOVF.py

[ubuntu-server-builder]

Launchpad user Ryan Harper(raharper) wrote on 2019-07-24T17:48:00.227924+00:00

You're quite right. I don't understand how this is expected to work.

Removing the customization file is going to prevent cloud-init from knowing it's on the same instance if the instance-id came from it.

but I don't think it did, I see this:

def read_vmware_imc(config):
@@ -296,6 +330,9 @@ def read_vmware_imc(config):
if config.timezone:
cfg['timezone'] = config.timezone

• Generate a unique instance-id so that re-customization will

• happen in cloud-init

• md['instance-id'] = "iid-vmware-" + util.rand_str(strlen=8)

Which is completely broken w.r.t keeping the same datasource. The comment suggests that when redeploying to the same instance, "new" customization didn't happen, so to work around that
the datasource generates a new "random" instance-id, to avoid cloud-init's check on instance-id.

The OVF datasource is going to need quite a bit of work to sort out which paths are working and which are not; having the platform provide a consistent set of metadata to the instance in a reliable form for the life-time of the instance is a requirement. This doesn't appear to be happening via the Imc path.

[ubuntu-server-builder]

Launchpad user Pengpeng Sun(pengpengs) wrote on 2019-07-25T03:01:01.924285+00:00

Hi Ryan,

Thanks for the investigation.

There is no consistent user data provided to OVF datasource, the "VMware Customization Config File" is available only when the customization happens on the VM.

1. Do Guest OS Customization on the VM (firstboot finds OVF Customization)
   With customization happened on VM, Open-VM-Tools can find and then copy cust.cfg from ESX server to the VM path /var/run/vmware-imc/, OVF Datasource will find the /var/run/vmware-imc/cust.cfg and then load it.

2019-07-24 07:50:48,823 - DataSourceOVF.py[DEBUG]: Found VMware Customization Config File at /var/run/vmware-imc/cust.cfg

2. Reboot VM (Second boot of same instance does not find metadata from DataSourceOVF)
   This time without customization happened on VM, so no cust.cfg will be found by Open-VM-Tools, OVF Datasource can not find /var/run/vmware-imc/cust.cfg, and then existing without datasource.

2019-07-24 07:58:19,856 - util.py[DEBUG]: waiting for configuration file took 90.110 seconds
2019-07-24 07:58:19,858 - DataSourceOVF.py[DEBUG]: Did not find VMware Customization Config File

As far as I know, there are 2 paths.

1. customize VM by cloud-init and then re-customize VM by cloud-init.
   This path works now since "random" instance-id keep cloud-init to search userdata for OVF datasource every boot

2. customization VM by cloud-int and then just reboot
   This path is broken, the expected behavior is at the second reboot, cloud-init keep network configuration unchanged.
   I do not think up a good solution, please shed some light on how to achieve it.

[ubuntu-server-builder]

Launchpad user Pengpeng Sun(pengpengs) wrote on 2019-07-25T03:03:55.076063+00:00

Links to VMware bug# 2377600

[ubuntu-server-builder]

Launchpad user Ilia(ilia-zenkevich) wrote on 2019-08-01T10:16:59.223605+00:00

What does it mean? "Links to VMware bug# 2377600" - can't find bug with such id. Can you please give a direct link?

[ubuntu-server-builder]

Launchpad user Pengpeng Sun(pengpengs) wrote on 2019-08-01T10:34:24.327741+00:00

@Ilia, It should be "Links to VMware INTERNAL bug# 2377600". It just a tracker for me.

[ubuntu-server-builder]

Launchpad user Eduardo Otubo(otubo) wrote on 2019-08-16T11:43:26.287500+00:00

Blocks Red Hat bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1593010